Retrieve, Read, Organize, Append and Split Adjust Financial Data

In this project, the challenge will not be simple and will require great programming skills (ranging from Data Mining to API building) and great financial and mathematical knowledge. DO NOT send a proposal if you think you are not prepared for it. If you think, otherwise, that you do have those skills, I strongly encourage you to send your proposal. This project will be organized into 5 steps: 1. Retrieve Data from a Website i) You'll have to create a program to download a file from the following website (Bovespa's Website, which is the greatest stock exchange in Brazil): [login to view URL]{{YEAR}}.ZIP ii) Download Data for each year, since 1986 to 2015. In order to do this, change the "{{YEAR}}" for the numeral. For example: In order to download the data for year 2015, the download link will be: [login to view URL] iii) You'll notice that it opens a CAPTCHA when you open this website. So, yes, the first task you'll need to do is bypass this captcha. iv) The file downloaded contains a great amount of financial data from Brazil's Stock Market for the year of 2015. But it is in .txt format, which leads us to the next challenge: Read the data. 2. Read a .txt Data inside the Downloaded File i) Each line of the .txt file corresponds to a security End-of-Day Data ii) The layout for reading the file can be found in the following link: file:///Users/vfssantos/Downloads/[login to view URL] iii) Use the layout above to read the data in a comprehensive way. iv) After being able to read the data, many lines of data that are not so useful. That leads to the next step which is Organizing the Data 3. Organize the data a. Each downloaded file will contain different data from different markets: i) Cash ii) Exercise of Call Options iii) Exercise of Put Options iv) Auction v) Factionary vi) Term vii) Forward with gain retention viii) Forward with continuous movement ix) Call Options x) Put Options b. Organize data by security name. c. Create a JSON object named by the security's name, and that contains the following data: i) Security Ticker ii) Company Name iii) Market Type iv) CODISI v) CODBDI vi) Data d. The Data in ([login to view URL]) corresponds to another JSON object containing, for each date: i) Open Price ii) High Price iii) Low Price iv) Close Price v) Number of trades vi) Number of contracts vii) Total Volume of contracts viii) Strike Price (if available) ix) Maturity Date (if available) x) Dividend xi) Split 4. Append the data: i) After having the data organized for each year, it is necessary to append every year's data. 5. Split-adjust the data: i) Here's one key challenge. You'll have to create an algorithm to detect when split-adjustment happen and also their magnitude. 6. After all this, you'll need to send this data to an API in the format specified in (.3). This API will interact with the database and will store the data. PS.: This is a part of a bigger project. And we AIM to change how the financial markets are here in Brazil. If you have the necessary skills and WANT to CHANGE peoples lives for better, I REALLY recommend that you send your proposal. (see the mind map here: [login to view URL]) The correct layout file*:[login to view URL]