Hi there,
I am Ahmed a Python developer, I have read yoyr project requirements and I understand that you are facing a probelm with your process getting killed automatically by the server.
I could solve that, and we will make a cron routine for that crawler...
contact me over chat, and we will talk abouth the details ...