We are looking for an experienced person to fulfilL the position of Data Engineer.
- Design and implement data pipelines to integrate large amounts of data from (to) many diverse storage systems
- Assemble large, complex data sets that meet functional/non-functional business requirements
- Build the infrastructure required to develop and test used technologies and algorithms
- Identify, design and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing the infrastructure, etc.
- Make data used by business analytics and machine learning
- Share knowledge by clearly articulating results and ideas to team members, potential data users and key decision makers
- Bachelor’s Degree in Computer Science or a related field
- At least 1 year of experience as a Data Engineer
- Experience with complex distributed systems and service-oriented architectures
- Experience with programming languages (C#/ Python/ Java/ Scala/ C/ Lua)
- Knowledge of data tech stack (HDFS, Docker, Kubernetes, ETL, Google BigTable, Spark, Hadoop, Column store/Row store)
- Experience with relational databases (SQL, MySQL, Hadoop, etc.)
- Experience with Data Hub / Lake / Warehouse
- Experience with Hive/Pig/ ETL batch/data pipe
- Familiarity with functional programming
- Familiarity with metadata management
- Experience with using Cloudera is a plus
- Experience with CI/CD tools and production code deployments is a plus
- Familiarity with DB horizontal scaling, SLA is a plus.