Find Jobs
Hire Freelancers

Database engineer to set up living cloud database and api for (near-)real time data delivery

€1500-3000 EUR

Cerrado
Publicado hace casi 2 años

€1500-3000 EUR

Pagado a la entrega
Requirements - Communicative: This is a (partly) collaborative project. At the end of the project, I need to have a decent understanding of how everything works. Communication and documentation will thus be important. - Terraform, SQL and Python programming. - Google Cloud Platform skills: it seems like google is the best option to host the databases on so you would need to have a firm understanding of the google stack (preferably with gcp data engineering certification) - (Optional: React front-end experience) What I am trying to build (NOTE that this is the entire vision, not all of this must be delivered in this project. This project is just to set up the foundation to enable me to build on): I have been trying to build the infrastructure to do academic research on a large volume of economic time-series data. The research would be both nowcasting of economic health and analyses of historic data. So for this, we would need to have a database that is constantly kept up-to-date from different data-sources and that can serve data to researchers. Additionally, we want to be able to integrate the analytics and models that we develop into the API to serve up the results of the models in a timely manner. I am currently setting up this research stream with a very small team but we would already like to make it scalable, and affordable for the long-term. Eventually (NOT IN THIS PROJECT) I would like to add a front-end so that I can provide access to the data and our model-outputs to less technical researchers. This does not need to happen in this project but the foundations should be built with this in mind. High-level idea of the needed elements (not rigid on this and open to suggestions): - a BigQuery database that would hold the full archive of data - cloud-based jobs that constantly pull new data from the source APIs and push it to the BigQuery - a python-based API framework built to expose the data and model outputs through api end-points. - cloud-based jobs to run multiple models on the data in the background and generate and save their outputs to a table in the bigquery Work that has already been completed: - Data extraction scripts in Python for retrieval of the data from the target sources - scripts to clean and transform the raw API data to the format I want in the BigQuery - 800+gb of raw data retrieved in Parquet format - SQL database schema draft (in SQLAlchemy) - First draft of the API written in Fast API Work that needs to be completed: Phase 1 - Tell me your hobby so I know you read this ad completely. - Review SQL database schema to identify where improvements can be made to the tables for the envisioned use-cases - Creation of BigQuery table to store historic data - partitioned by day - Terraform (or some other language) file to manage infrastructure changes and service accounts / permissions - Github repositories to manage source code for each of - images that get built (e.g source code in src folder and docker file) - terraform file(s) - Containerise (docker) the data extraction scripts to be stored in good cloud container repository - Containerise (docker) the Fast API app to be stored again in GCP container repository - Deploy an instance of the Fast API app on cloud run - Deploy appropriately sized VMs to run the containers with the data extraction scripts Phase 2 - upon successful completion of Phase 1 - Add front-end to enable others to explore the data interactively (basic graphs and tables) As said in the beginning phase 2 is not necessary to be completed in this project but if you can help with this as well, or can involve someone who you can collaborate with to do so, the budget can reflect this. Send your resume and envisioned approach and I hope we can work together. (I plan on reaching out to plan the interviews the week of the 23th of May) Kind regards, Milan
ID del proyecto: 33688295

Información sobre el proyecto

24 propuestas
Proyecto remoto
Activo hace 2 años

¿Buscas ganar dinero?

Beneficios de presentar ofertas en Freelancer

Fija tu plazo y presupuesto
Cobra por tu trabajo
Describe tu propuesta
Es gratis registrarse y presentar ofertas en los trabajos
24 freelancers están ofertando un promedio de €2.623 EUR por este trabajo
Avatar del usuario
Hey, my hobby - development :) It is also my work. I got acquainted with everything really, but the question is why not use AWS? I would suggest the following stack: 1. AWS RedShift database for quick access and AWS Athena/S3 or BigQuery for big data (dependence from goal) 2. Terraform use for DevOps 3. Bitbacket for source code / Bitbacket Pipeline for CI/CD 4. Can run task serverless use AWS lambda, AWS Fargate 5. CloudFront for UI 5. Slack for notifications best regards, Vasko
€3.000 EUR en 20 días
4,9 (37 comentarios)
6,3
6,3
Avatar del usuario
Hello_________________________________________⭐ I read the whole concept of the project and I am very interested in your project. I can do it well depending on your requirements. My skill, creativity, effort and everything else will be served for you. I will do my best to finish it on time in a very high quality level and satisfy you. Thank you for clicking my bid. Hope to hear from you soon.
€2.250 EUR en 7 días
5,0 (9 comentarios)
5,6
5,6
Avatar del usuario
Hi there, I am Rashid, Hope you are doing great. Firstly about the Database engineer to set up living cloud database and api for (near-)real time data delivery that I have a great experience in it. I am very confident to pull it off once awarded. I am a Full stack developer with a team of experienced developers and designers on my side, together we have 5+ years of experience in API, SQL, Terraform, Google Cloud Platform and Python I aim to establish a long term working relationship with my clients. It would be my honor to have you on board with us. Please let me know what is needed to secure the bid! Thanks for your consideration. Rashid Amjad.
€2.500 EUR en 15 días
5,0 (1 comentario)
2,0
2,0
Avatar del usuario
Hello! I am a proved Javascript developer with 8 years of background and familiar with skills including SQL, Google Cloud Platform, Terraform, API and Python. Years of experience let me handle the project with well-organized structure and optimized which provide fast loading/processing speed. I can work full time and be available during your working hours. I would like to reassure you that I can do great job and deliver the result of good quality. I am looking forward to working with you. Thank you.
€2.500 EUR en 7 días
0,0 (0 comentarios)
0,0
0,0
Avatar del usuario
I have carefully read your job description and very interested in your project. I am sure I am very suitable for it. I am a senior FullStack developer and I have 8+ years in this field. I am well versed in React(Next), Angular, vue(Nuxt), PHP(Laravel, CI), WordPress, CSS, JQUERY, Python, Django, Node.js, Express and have rich experiences of making such projects. And I have also strong fundamental about c++ and Python. Responsibility is my best advantage. Reading the project description, I thought I can do it very well. I want to discuss about the job in detail via chat. I have worked on similar projects to what you are looking for, and I am confident I can exceed your expectations. I am always here, so please feel free to contact me. Best regards. Filipnicol.
€2.500 EUR en 20 días
0,0 (0 comentarios)
0,0
0,0
Avatar del usuario
I can manage your database server for long time. Supporting database: Firebase/Google Cloud SQL Looking forward to working with you. Best ergards
€2.000 EUR en 20 días
0,0 (0 comentarios)
0,0
0,0
Avatar del usuario
I am professional developer in python, postgresql , GCP, Bigquery, GIt. I am very interested in this project. -As a Database engineer, I can improve the speed using index and pk/fk -I have experience in this because I have completed this job last time with my client -I have experience in this , I have used elastic search for big data -I have strong experience in Git - images that get built (e.g source code in src folder and docker file) I can manage this job. - Creation of a data extraction script to remove data from the SQL database which is 30 days old and push to BigQuery (envisioned to run daily to remove previous day’s data push to bucket/bigquery then remove from SQL db) I have rich experience ETL and elastic search - Containerise (docker) the data extraction scripts to be stored in good cloud container repository Docker environment is my favoriate work place - Containerise (docker) the Fast API app to be stored again in GCP container repository I can do this job -Using python I have built fast api last time on my last elastic project -ETL project makes me strong in this job. Phase 2 - upon successful completion of Phase 1 - Add a caching layer to the API services (whether that’s redis, or local in-built caching on Fast API if that is sufficient) - Deploy / run ML models I build on GCP leveraging serverless / cloud solution where possible I can complete this job successfullly
€3.000 EUR en 7 días
0,0 (0 comentarios)
0,0
0,0
Avatar del usuario
Hi, It is easy, I can do on time. I work online, where you can track progress of your project. I have 6 years of experience in development(websites, web applications, mobile apps, desktop applications, I/UX), using PHP, WordPress, Java, Python, JavaScript, ReactJs, Bootstrap, I can start now, please come to chat, to discuss more.
€1.500 EUR en 2 días
0,0 (0 comentarios)
0,0
0,0

Sobre este cliente

Bandera de BELGIUM
Linho, Belgium
0,0
0
Forma de pago verificada
Miembro desde may 13, 2022

Verificación del cliente

¡Gracias! Te hemos enviado un enlace para reclamar tu crédito gratuito.
Algo salió mal al enviar tu correo electrónico. Por favor, intenta de nuevo.
Usuarios registrados Total de empleos publicados
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Cargando visualización previa
Permiso concedido para Geolocalización.
Tu sesión de acceso ha expirado y has sido desconectado. Por favor, inica sesión nuevamente.