Distributed crawler / crawling system / Web mining project

You will (or already have) 1) write a distributed crawler system running multiple crawlers with a command an control (in PHP or Java or Node.JS) 2) eventually crawl over 160M pages of a social network ([login to view URL]) and defeat potential crawling roadblocks 3) deliver results in a text comma delineated file(s) based on our data format You will get: -Massive cloud computing accounts. There will be almost no limits. -VK Account Directory -help from our Engineering team. You will get paid along the way... so don't worry about that. And this will be fun.