Configure HDFS and Spark on DC/OS Cluster

€18-36 EUR / hour

Cerrado

Publicado

hace más de 5 años

€18-36 EUR / hour

We are running DC/OS Cluster on AWS, and manage it using Terraform. We have tried to deploy HDFS on DC/OS, but we still can't connect from locat to cluster's HDFS, and also can't consume the data from HDFS/ S3 using Spark, due to error on the miss of hadoop client. HDFS Cluster and Spark are set up and show healthy states. Client's [login to view URL] and [login to view URL] are also downloaded and placed under local hadoop installation, but. 1) 'hdfs dfs -ls /' lead to error 2) spark complains on problem to find hadop / hdfs Here are the links on the very same problems, but we haven't managed to resolved them. So we're looking for expert help that had already experience on running HDFS / DCOS, so point out where to change. Thanks [login to view URL] [login to view URL] Please answer for below questions when apply to job 1. Do you have experience with DC/OS installs and maintenance? 2. Do you ever configure DC/OS HDFS and DC/OS Spark 3. Have you managed custom Spark deployment (--packages, s3://) 4. Do you have suggestions to make this project run successfully? 5. What questions do you have about the project?

ID del proyecto: 18324290

Información sobre el proyecto

10 propuestas

Proyecto remoto

Activo hace 5 años

¿Buscas ganar dinero?

Dirección de email

Beneficios de presentar ofertas en Freelancer

Fija tu plazo y presupuesto

Cobra por tu trabajo

Describe tu propuesta

Es gratis registrarse y presentar ofertas en los trabajos

10 freelancers están ofertando un promedio de €30 EUR /hora por este trabajo

@emicsoft211

Hello, I can Configure HDFS and Spark on your dc cluster. Please let me know if you are interested so we can discuss regarding the project over chat. Waiting for your reply. Thank you

€36 EUR en 40 días

4,8

(25 comentarios)

4,7

@dirisalagopal

I am working as a software developer with 10+ years of experience. Industry experience includes Big Data, Backend, DevOps (AWS and Azure) and etc. I did my MTECH in the Department of Computer Science Engineering, Indian Institute of Technology (IIT) , Guwahati, India. Some of the skills are mentioned below. Amazon Web Services: EC2, Load Balancers, RDS, Elastic Beanstalk, Athena, Glue, EMR IAM, VPC, Security Groups, Cloud Formation, Lambda, S3, CLoudFront, Redshift, Route 53, Cloud Watch, Kinesis, SNS, SQS, API Gateway and etc. Azure: VMs, SQL Databases, Storage and etc. Programming Languages: C, C++, Java, Go, Python, Bash Big Data Technologies: Hadoop, Spark, Hive, HDFS, HBase, Kafka, Flume, Memsql, Aerospike, Redis and etc. Miscellaneous: Algorithms, Data Structures, Distributed Systems, Operating Systems Databases: Oracle, MySQL, SQL Server Operating Systems: Windows, Linux, Ubuntu and etc. Code Deployments: Puppet, Docker, Airflow

€30 EUR en 15 días

4,9

(28 comentarios)

4,8

@deytps86

Hello I am working in Bigdata/Hadoop technologies for years. I worked in Hadoop, Mapreduce, Spark (Streaming & MLLib), Kafka, ELK Stack, Cassandra, MongoDB, Postgre. I implemented above using Java Scala and Python. Have worked Shell, Perl and Python Can we talk further on this? Thank you!

€30 EUR en 40 días

4,9

(22 comentarios)

4,9

@AmolZinjadeP

HI, I have more than 4+ years of experience in hadoop technologies like HDFS, MapReduce, Spark, Sqoop etc. Contact me for more detail

€33 EUR en 40 días

4,6

(7 comentarios)

3,7

@dineshrajputit

Hi, I have 7 years of experience and working on hadoop, spark, nosql, java, BI tools(tableau, powerbi), cloud(Amazon, Google, Microsoft Azure)... Done end to end data warehouse management projects on aws cloud with hadoop, hive, spark and presodb. Worked on multiple etl project like Kafka, nifi, flume, mapreduce, spark with XML/JSON., Cassandra, mongodb, hbase, redis, oracle, sap hana, ASE.... Many more. Let's discuss the required things in detail. I am committed to work done and strong in issue resolving as well. Thanks

€20 EUR en 40 días

4,9

(3 comentarios)

2,0

@findjramj

I can help you with this project: - Well Experienced in Big Data (Hadoop | Kafka | Spark | NoSQL | Cloud) Administration and Platforms to Accommodate the Expanding Business Needs. - Well Experienced in Different Vendors (Hortonworks, Cloudera, MapR, etc) of Big Data and Cloud. - Big Data Capacity Planning, Tuning, Solution Architecting, Issue Resolution, and Security. - Big Data Analytics Solution Architecture, and its Implementation. Answers: 1. Do you have experience with DC/OS installs and maintenance? Yes 2. Do you ever configure DC/OS HDFS and DC/OS Spark Yes 3. Have you managed custom Spark deployment (--packages, s3://) Yes 4. Do you have suggestions to make this project run successfully? Once I get more insight into the use case, I can guide accordingly 5. What questions do you have about the project? Once I get more insight into the use case, I might have more questions Thanks, JJ

€30 EUR en 40 días

0,0

(0 comentarios)

0,0

@CanvasInfotech10

Hello, Hope you are doing absolutely great. Instead of writing Proposal I would like to approach you directly for this job, as we have been working on the AWS since last 3 years. Let's take this forward as it looks like you need of an AWS expert to work on running HDFS / DCOS, so point out where to change. Hours of work: 40 Hr/week We have a working experience in node.js, MongoDB, Express JS, NoSQL Couch & Mongo, PostgreSQL, webRTC, MySQL, PHP, System Admin, frontend and backend development, load balancing, Video Broadcasting, Video Services, Amazon Web Services, Cloud Computing, Linux, Network Administration, System Admin, Database Management. So, here I’m confident that we will able to work on your project. Looking for to connect with you in chat. Best Regards, Akash!

€20 EUR en 40 días