I am looking a ASP.NET application (VB.net preferred, C# ok too) that can extract information 'objects' from a page. Basically, I want a spider that can crawl several of my competitors web sites and retrieve current prices on different products on a regular basis. The only data that I am looking for from each product page is the product name, the price, the URL, and the meta name="keywords" section if available. This data will be written to a CSV file. It must be easy to configure the search strings so that I can add a new site (i.e. look for "Our Price: " or "Your Price: "). It should very stable and should run as fast multithreaded, automated spidering engine.
1) Complete and fully-functional working program(s) in executable form as well as complete source code of all work done. 2) Complete ownership and distribution copyrights to all work purchased.
Windows 2000. ASP.net (VB preferred)