Financial Data Engine - for ATLASFSD - open to bidding

Cancelado Publicado hace 7 años Pagado a la entrega
Cancelado Pagado a la entrega

This application will crawl several financial data websites every 6 hours:

1) [login to view URL]

2) [login to view URL]

3) [login to view URL]

Scraping program should be written In Python, probably Scrapy or similar.

Data to be scraped will be organised as:

A) Providers - various types of financial institutions that will have various characteristics including a list of products as well as various other fields describing the company

B) Products - there are several different categories of products. There are also lots of fields associated with each product, and there will be several products sold by each provider in each category.

A freelancer will to design and implement the MySQL Schema for all of the data above, including all of the various fields to match the scraped data as well as the relations between the tables. This will be based on the format and relations in the data from the three sources above, and will have input from me.

The raw data will be put through several programs:

1) Check the scraped data has worked correctly (specific format for each field)

2) If error, then the product (and the changed field) will be listed in an 'Errors' list in the GUI for the user to check the scraper and adjust inputs to fix

3) If the value has changed then the product (and changed field) will be listed in a 'Changes' list in the GUI for the user to track changes in data scraped

GUI with 4 x different summary tables, each table to have search, filter and sort functions, and 2 x summary pages (product and provider):

1) List of providers and some summary details

2) List of products and some summary details

3) List of changes to the data since the last scrape (not scraper errors)

4) List of potential scraper errors to be resolved (data error or scraped value much different format to the last scrape based on rules for each field)

GUI to have a number of functions that perform several database queries and return the results (if any) to one of the summary pages:

1) List of providers and a set of specified summary details

2) List of products and a set of specified summary details

3) List of changes to the data since the last scrape (not scraper errors)

4) List of potential unresolved errors in recent scrapes (not just changes in field value)

Engine to create a fourth set of data recording every provider and product:

- Three data sources to be triangulated based on specific criteria that I will specify

- Functionality to change source of data for each product or provider

HTML MySQL PHP Arquitectura de software Extracción de datos web

Nº del proyecto: #12839702

Sobre el proyecto

22 propuestas Proyecto remoto Activo hace 7 años

22 freelancers están ofertando un promedio de $2440 por este trabajo

r4rony

Hello Client Greetings..........!! We are web based Development Company. We have been nominated in Top 100 web Vendors 2013. We have experts available on various technologies Magento, Wordpress, Joomla, Drupal, Más

$2371 AUD en 30 días
(185 comentarios)
8.5
webbookstudio

Hello, my name is Olya. I represent Ukrainian IT Company «Webbook». We provide website design and web and app development services for organizations, public and government institutions, company or private web-pages. Más

$2500 AUD en 25 días
(178 comentarios)
8.3
Shopify

Hello, I want to show you all relevant Demo and Designs which is similar to your project completed previously. To make sure about the requirement set and customizations, I want to discuss this project with you further Más

$2577 AUD en 30 días
(30 comentarios)
7.2
esalem

Hello, I am VB, VBA, C#, WPF, Java, Delphi, PHP, ASP.NET, CAD and GIS developer, I developed many desktop applications and custom tools, macros and Add-Ins. For reference see my profile. I am ready to discuss and Más

$4789 AUD en 40 días
(26 comentarios)
7.2
mmadi

Dear Client, Greetings from Flowgica technologies, I have experience with these skills. We do have similar experience doing something similar to yours therefore I am looking forward to discuss more about more details a Más

$2100 AUD en 36 días
(43 comentarios)
7.0
samitalways4u

Hello , I read your job descriptions very well and I am glad to know you that I can do it. Please talk to me then we can discuss more in detail about this project. Please Give me Chance to work with You. Thank You..

$2105 AUD en 30 días
(111 comentarios)
7.5
Beautistar

Dear client, I am a professional mobile app & web server developer with 6+ rich experience. I would like to work on your project and deliver the project. I have developed many applications such as social app, dati Más

$2500 AUD en 30 días
(22 comentarios)
6.6
mehedi276

"Rate and duration will be fixed on discussion" Dear Hiring Manager, I have gone through your job posting and become very much interested to work with you. I am an expert in these fields. I have already completed sev Más

$2319 AUD en 30 días
(59 comentarios)
5.6
shahiddar

Hello, My name is shahid from Kashmir Over the last 7 years, I have worked for several clients. Joined Freelancer with over 7 years rich experience in the field.I have successfully completed more than 1000 projects in Más

$1500 AUD en 0 días
(15 comentarios)
5.1