Indexer / Spider / Robot

Hi, The project that you have posted, is my day to day work , reviewing and tweaking the python scripts is what my current job. I have been working as an Script automation engineer in python and Perl for past 4 years. I have an out and out experience in writing complex automation frameworks in python and Perl.I have been awarded Live wire from my Client manager for the work I have done. I'm an open source contributor involved in framework development, you can google framework name as PIKE (its an EMC Isilon project). My plan to complete this project is divide the project into smaller chunks(modules) and understand thoroughly what exactly the project is all about and do it in a best possible way. Please contact me. Regards, Govardhan Yadav

Hello! This is an interesting project and I would like to review the details with you. I have one question. The bot will crawl and index MP3 from all over the internet but you say you have an exact source for that. Can you kindly explain the meaning of it? As for my experience, I'm arrived here from ELance (another freelancer website) where I did scrapers/bots and crawlers for the most various subjects. I'm proeficient with analysing and programming my bots to every situation, so they can adapt and auto-mutate themselves for better functionalities. My main programming language is Python, which I have over five years of experience. I'm confortable with using most of the libraries available and coding my own. I would gladly want to hear more details about this project. With my best regards, Tiago

Hello employer, I have much experience in creating custom crawlers, a dirty work indeed. I can solve your problems in 5 days (through planning with you; writing; testing; deploying). I plan to develop everything in PHP (crawling with other languages can become painful), PHP has many tools to deal with page crawling (top one is DOMDocument which is really fast). Anyway I might require more time to deal with this problem, depending on how many sites you need to crawl (since each crawler is diferent). Also, to get a glance of architecture I would like to build a "center" crawler to handle all the work, and add little (1 file crawling script) classes as plugins to the main one. In this way you will be able to control each crawler differently. Since you asked also for a "Shell script", I can make you able to command the crawler I will be building under CLI (you will be needing a PHP builtin server (so version 5.3 or higher)). Also (the most difficult part) we can obtain ID3 information just by downloading a few bits. Regards and feel free to contact me for whatever you need.