已申请 实习 应届毕业生 & 经验工作者 最新工作
Senior Data Engineer (PySpark, Python and SQL)
- 2023-12-15
-
Singapore, Singapore
Senior Data Engineer (PySpark, Python and SQL)
职位详情
Roles & Responsibilities
Tell employers what skills you have
PySpark
Airflow
Apache Spark
Kubernetes
AWS
ETL
Information Technology
EMR
Apache Hadoop
Python
The Senior Data Engineer will be responsible for the design, develop, and maintain:
- Strong Experience on building Data Pipelines using Spark hosted on AWS EMR or Glue.
- Batch Pipeline Orchestration using on Apache Airflow.
- Deploy jobs on Kubernetes (AWS EKS).
- Code Repo Pipeline to automate Continuous Integration and Continuous Deployment (CI / CD).
- Structure Tables with Partitioning and Clustering to increase Cost & Performance Benefits.
- Guide Data Analysts and Data Scientists to write efficient queries and workloads.
- Data Sharing with On-Demand Encryption/Decryption which can operate at Scale.
- Running Containerized ETL workflow at scale.
Qualifications
- Bachelor Degree of Computer Science or IT
- At least 4 years of Data or Software Engineering experience
- Have strong fundamentals in Computer Science concepts like Cloud Computing Architecture, Distributed Computing, High-Velocity Data Processing, Lambda Architecture, etc…
- Strong Data modeling and managing Distributed Computing Platforms for Data Processing.
- Have at least 3+ years of ANSI SQL and writing resource-efficient queries.
- Have at least 2+ years of professional programming experience in PySpark.
- Have a good understanding of how Kubernetes clusters work and scale on-demand.
- Have adequate experience using Containers for Data Engineering workload.
- Implemented manual or automated tools for Data Quality, Catalog, and Lineage.
- Uphold the sense of Frugality across Data Engineering teams.
- Have Good Interpersonal and Presentation Skills to explain and promote Best Practices across the organization with both technical as well as non-technical stakeholders.
Tell employers what skills you have
PySpark
Airflow
Apache Spark
Kubernetes
AWS
ETL
Information Technology
EMR
Apache Hadoop
Python
小心骗局。不要向不明来源提供个人信息或付款。在采取行动之前验证身份。立即举报任何疑似骗局。保持警惕,保持安全。
INCOME INSURANCE LIMITED
© Copyright 2024 Agensi Pekerjaan JEV Management Sdn. Bhd., registered in Malaysia (Company No: 201701016948 (1231113-U), EA License No. JTKSM860)
© Copyright 2024 Job Majestic Sdn. Bhd., registered in Malaysia (Company No: 201701037852 (1252023-X))
版权所有