Roles & Responsibilities

The Senior Data Engineer will be responsible for the design, develop, and maintain:

• Strong Experience on building Data Pipelines using Spark hosted on AWS EMR or Glue.
• Batch Pipeline Orchestration using on Apache Airflow.
• Deploy jobs on Kubernetes (AWS EKS).
• Code Repo Pipeline to automate Continuous Integration and Continuous Deployment (CI / CD).
• Structure Tables with Partitioning and Clustering to increase Cost & Performance Benefits.
• Guide Data Analysts and Data Scientists to write efficient queries and workloads.
• Data Sharing with On-Demand Encryption/Decryption which can operate at Scale.
• Running Containerized ETL workflow at scale.

Qualifications

• Bachelor Degree of Computer Science or IT
• At least 4 years of Data or Software Engineering experience
• Have strong fundamentals in Computer Science concepts like Cloud Computing Architecture, Distributed Computing, High-Velocity Data Processing, Lambda Architecture, etc…
• Strong Data modeling and managing Distributed Computing Platforms for Data Processing.
• Have at least 3+ years of ANSI SQL and writing resource-efficient queries.
• Have at least 2+ years of professional programming experience in PySpark.
• Have a good understanding of how Kubernetes clusters work and scale on-demand.
• Have adequate experience using Containers for Data Engineering workload.
• Implemented manual or automated tools for Data Quality, Catalog, and Lineage.
• Uphold the sense of Frugality across Data Engineering teams.
• Have Good Interpersonal and Presentation Skills to explain and promote Best Practices across the organization with both technical as well as non-technical stakeholders.

Tell employers what skills you have

PySpark
Airflow
Apache Spark
Kubernetes
AWS
ETL
Information Technology
EMR
Apache Hadoop
Python