Sr Data Engineer

Responsibilities:

Design, Develop, and Maintain Data Pipelines:

Utilize Databricks and Spark, along with other cloud technologies as needed.

Optimize data pipelines for performance, scalability, and reliability.

Ensure Data Quality and Integrity:

Maintain data quality and integrity throughout the data lifecycle.

Collaborate with Stakeholders:

Work closely with data scientists, analysts, and other stakeholders to understand and meet their data needs.

Troubleshoot and Resolve Data Issues:

Identify and resolve data-related issues.

Provide root cause analysis and recommendations.

Documentation and Communication:

Document data pipeline specifications, requirements, and enhancements.

Communicate effectively with the team and management.

Data Validation and Analysis Tools:

Create new data validation methods and data analysis tools.

Share best practices and learnings with the data engineering community.

Implement ETL Processes and Ensure Compliance:

Implement ETL processes and data warehouse solutions.

Ensure compliance with data governance and security policies.

Qualifications:

Bachelor's degree in computer science, Engineering, or related field, or equivalent work experience.

3+ years of experience in data engineering, preferably with Databricks and Spark.

Proficient in SQL and Python, familiar with Java or Scala.

Experience with cloud platforms, such as Azure or AWS.

Experience with big data technologies, such as Kafka, Hadoop, Hive, etc.

Familiarity with data warehouse and data lake concepts and architectures.

Experience with data integration and ETL tools, such as Azure Data Factory or Talend.

Familiarity with data visualization and reporting tools, such as Power BI or Tableau.

Strong analytical and problem-solving skills.

Excellent communication and teamwork skills.