Financial Data Engine - for ATLASFSD - open to bidding
$1500-3000 AUD
Pagado a la entrega
This application will crawl several financial data websites every 6 hours:
1) [login to view URL]
2) [login to view URL]
3) [login to view URL]
Scraping program should be written In Python, probably Scrapy or similar.
Data to be scraped will be organised as:
A) Providers - various types of financial institutions that will have various characteristics including a list of products as well as various other fields describing the company
B) Products - there are several different categories of products. There are also lots of fields associated with each product, and there will be several products sold by each provider in each category.
A freelancer will to design and implement the MySQL Schema for all of the data above, including all of the various fields to match the scraped data as well as the relations between the tables. This will be based on the format and relations in the data from the three sources above, and will have input from me.
The raw data will be put through several programs:
1) Check the scraped data has worked correctly (specific format for each field)
2) If error, then the product (and the changed field) will be listed in an 'Errors' list in the GUI for the user to check the scraper and adjust inputs to fix
3) If the value has changed then the product (and changed field) will be listed in a 'Changes' list in the GUI for the user to track changes in data scraped
GUI with 4 x different summary tables, each table to have search, filter and sort functions, and 2 x summary pages (product and provider):
1) List of providers and some summary details
2) List of products and some summary details
3) List of changes to the data since the last scrape (not scraper errors)
4) List of potential scraper errors to be resolved (data error or scraped value much different format to the last scrape based on rules for each field)
GUI to have a number of functions that perform several database queries and return the results (if any) to one of the summary pages:
1) List of providers and a set of specified summary details
2) List of products and a set of specified summary details
3) List of changes to the data since the last scrape (not scraper errors)
4) List of potential unresolved errors in recent scrapes (not just changes in field value)
Engine to create a fourth set of data recording every provider and product:
- Three data sources to be triangulated based on specific criteria that I will specify
- Functionality to change source of data for each product or provider
Nº del proyecto: #12839702
Sobre el proyecto
22 freelancers están ofertando un promedio de $2440 por este trabajo
Hello, my name is Olya. I represent Ukrainian IT Company «Webbook». We provide website design and web and app development services for organizations, public and government institutions, company or private web-pages. Más
Hello , I read your job descriptions very well and I am glad to know you that I can do it. Please talk to me then we can discuss more in detail about this project. Please Give me Chance to work with You. Thank You..
Dear client, I am a professional mobile app & web server developer with 6+ rich experience. I would like to work on your project and deliver the project. I have developed many applications such as social app, dati Más