Screen scraping

The task is to create a windows console application in c# that will extract data, like screen scraping, from a web site and write the data down into two different files semi colon separated. The program should be possible to run multiple times and should then append data to the output files.

Every day the website posts a lot of new information, but it always comes in id sequence. So if the current post is something like [url removed, login to view], then the next post will be the same only with 1322 at the end instead of 1321.

The intention is to have the program running as a batch job and fetch all data that have been posted since last time the program ran. Therefore the program needs a configuration parameter with a field like LastStoredId.

So the program should start by looking at that id, say that it is 100. Then it should fetch all data from the site like this:

[url removed, login to view]

[url removed, login to view]

[url removed, login to view]

[url removed, login to view]

[url removed, login to view]

[url removed, login to view]

Then when the program tries to fetch id 106 it might be that you get a handled error from the site saying something like “Oops, the page you searched for was not found. Please go back and try again”. When that happens you know that there are no more information to fetch. Then you store the last retrieved id, in this case 105 in configuration parameter file as LastStoredId. When running the program again next day or next week, then there might be more data to fetch so that you store data for id 106, 107… and so on until you once again can’t find any more info.

The site to collect data from also has forms authentication. The program should somehow be able to sign with a username and password that should be configurable from the configuration parameters file where LastStoredId field is also stored.

So the suggestion for the configuration file is that you only have three rows. Row one has a single number that is the value for LastStoredId, line two contains username and finally line three contains password.

Programación en C#

Nº del proyecto: #8758682

Sobre el proyecto