Position Overview: We are seeking two experienced Senior Data Engineers to join our team. The ideal candidates will be responsible for managing and optimizing data pipelines, ensuring data security and integrity, and providing production support on Azure. This role requires expertise in Databricks, Azure Data Factory, and Azure functions, along with strong skills in Python and SQL.
Job Requirements:
- Production Support Experience: Proven experience as Production Support Engineers on Azure.
- Expertise in Databricks and Azure Services: Advanced skills in Databricks, Azure Data Factory, and Azure functions.
- Data Pipeline Management: Proficiency in managing data pipelines, ensuring performance, data security, integrity, and validation checks, with awareness of SLAs & SLOs.
- Programming Languages: Strong skills in Python and SQL.
- Databricks Workspace Management: Experience with Databricks workspace, clusters, and jobs.
- Monitoring and Alerting: Ability to monitor and alert on data pipeline failures, debug issues, perform root cause analysis, and address performance bottlenecks.
- Azure Data Lake Optimization: Skills in optimizing Azure Data Lake and capacity planning.
- Code Promotion: Experience with code promotion using DevOps tools.
Day-to-Day Responsibilities:
- Production Support: Provide production support for Azure environments, ensuring smooth operation and quick resolution of issues.
- Manage Data Pipelines: Oversee the management of data pipelines, ensuring high performance, security, and integrity, while adhering to SLAs & SLOs.
- Utilize Databricks and Azure Services: Expertly use Databricks, Azure Data Factory, and Azure functions to manage and optimize data workflows.
- Programming and Scripting: Write and maintain scripts in Python and SQL to support data engineering tasks.
- Monitor and Debug: Continuously monitor data pipelines, alert on failures, debug issues, perform root cause analysis, and address performance bottlenecks.
- Optimize Azure Data Lake: Plan and implement optimization strategies for Azure Data Lake, ensuring efficient capacity management.
- Code Promotion: Assist with code promotions using DevOps tools, ensuring smooth transitions and deployments.
This position pays 19-21/hr USD