RSS Feed Extractor

A Utility to Find All RSS Feeds in a Domain P9 Project General Information: P9 is the code name for an information based internet and social media start-up that has developed and proved the alpha version of the service and is now improving and adding components for the launch version. There are many components for this service and we will put certain definable components out for bid. We may also invite people who have completed these components to join the project on a permanent basis. Job Overview: Create a utility that identifies and extracts all of the RSS or similar feeds in a domain. Job: P9-Data Input Component-RSS feeds module-Extraction from a URL Technology: • Java 7 (not Java 8). • See Technology requirements below. If the design of the program requires additional software developed by third parties, the software must be open source and we must approve of the license(s) for any such software used in the program. Description of Work: Create a program that processes a list of URLs and finds for each URL all RSS, Atom and similar feeds reachable from the given domain but only in pages in the given domain. (In this document, “RSS” refers to RSS, Atom and any other similar feed/link that provides news story data). All classes shall reside in the package com.c2g2.p9.rss.web.crawler. In this package, provide a class called Main to drive the process, you can organize the code in this package as you see fit. Main takes the following command line arguments: -input a URL to the input CSV file (required) -output a URL to the output CSV file (required) -fail a URL to the failure output CSV file (required) -help prints command line help (optional) For each URL in the input file, the program crawls the whole site and looks for all RSS URLs. If there is a problem processing a URL from the input file, log an error, and continue to the next record. Unless you can prove otherwise, each URL that the program finds has to be read and fed to a RSS parser to determine if the link reads a valid RSS feed. The program can take shortcuts, for example, URLs ending in .html, .gif, .jsp and so on can be skipped. The list of file extensions to skip should be places in a [login to view URL] file in the class path and contain for example: skip-ext-list = .jsp .html .html .gif .jsp .png .jpeg .asp .aspx The program should smartly cache already visited URLs for a given domain to avoid fetching a URL over and over again. When the program finds a valid RSS feed, it should write an output CSV record with the following columns: 1. URL from input file 2. ISO-8601 timestamp in the combined date and time with UTC format. For example: 2014-03-21T00:47:06Z 3. RSS feed URL The column names are: 1. Domain 2. Timestamp 3. RSS When the program cannot reach a domain from the input file, it should write an output CSV record with the following columns: 1. URL from input file 2. ISO-8601 timestamp in the combined date and time with UTC format. For example: 2014-03-21T00:47:06Z The column names are: 1. Domain 2. Timestamp SEE ATTACHED DOCUMENT