- Database Design
- Shell Scripting
The Data Engineer in the Analytics department is tasked with creating, managing and transforming business-essential data to be used for analysis and prescriptive or predictive modeling. Data pipelines will also need to be created and maintained to support ongoing models, aiding in their continuous learning. Data engineers work closely with analysts and are largely in charge of architecting solutions for analysts that enable them to do their jobs.
Develop and maintain a verified analytics database.
Work closely with Analysts to prepare data to be used in predictive/prescriptive modeling.
Develop data pipelines to feed continuous learning models in a production environment.
Assist in transitioning data sources to the Azure cloud.
Integrate external data sources to enrich internal data.
Explore ways to enhance data quality and reliability.
4-6 years of experience in similar roles
Background in statistics and analytics
Technical expertise with data models and data mining
Experience in data pipeline and workflow management
Hands-on experience with SQL database design
Proficient in at least one of the following programming languages: Python, Shell Scripting, Scala, Java
Experience with Big Data tools, such as Spark, Kafka, Hadoop, HBase, etc.
Experience with tools for authoring ML workflows and pipelines, such as Airflow, Kubeflow, etc.
Experience workiing with containers and container orchestration solutions, such as Docker, Kubernetes, Mesos, etc.
Experience with Cloud platforms, such as AWS, Azure, Google, etc.
Familiarity with Linux
Familiarity with the DevOp concepts
Graduate-level degree or college graduate with Data Engineering Certificate