Extract CommonCrawl data (EC2, Cluster Operations) - great big data project for your CV

We'd like you to build a set of functions that allows us to work with Common Crawl on a monthly basis (or whenever a new crawl is released). [login to view URL] Here is an article that explains this for PHP or Python (see comment section). [login to view URL]@paulrim/mining-common-crawl-with-php-39e14082c55c Basically we need to get all crawl data on EC2 Ubuntu Spot Instance from the list of all files. We need to make sure the list (can be in a central database) is worked on fully and we can resume as spot instances do not live very long. We have a small PHP (or Python) script that parses HTML and extracts data (just like in the article). Once done next file should be worked on. We recommend periodic reboots for swap space and machine usage. We expect a set of tools that runs without interruptions and produces a folder in S3 with the parsed data. We will then get all the files in that folder and imported them (extract from S3 is not part of this project, we will cover this).