Data Engineer

Job Description:
• Job Title: Sr. Data Engineer

• Vendor Pay Rate:

• Hybrid

• Job Description: The ideal candidate will have a strong background in data engineering and development, with expertise in various data skills and tools. The role involves working on complex data projects, collaborating with cross-functional teams, and ensuring the efficient processing and management of data.
Key Responsibilities:

• Develop and maintain data pipelines using Python and Pyspark.

• Design and implement efficient SQL queries for data extraction and manipulation.

• Work with Aurora Postgres and Redshift for data storage and management.

• Utilize Informatica or other ETL tools for data integration and transformation.

• Implement and manage AWS cloud services, including Athena, S3, ECS/Docker, EMR, EC2, Lambda, CloudWatch, and EventBridge.

• Use Airflow or other orchestration tools for workflow management.

• Ensure data quality and integrity through rigorous testing and validation.

• Collaborate with other teams to understand data requirements and deliver solutions.
Required Skills:

• Proficiency in Python and Pyspark.

• Solid experience in AWS Glue

• Strong SQL skills.

• Experience with Aurora Postgres and Redshift.

• Knowledge of Informatica or other ETL tools.

• Experience with AWS cloud services, including Athena, S3, Lambda, CloudWatch, and EventBridge.
Preferred Skills:

• ECS/ Docker (or any docker experience/concepts)

• EMR, EC2 (or knowledge of any other distributed computing)

• Familiarity with Airflow or other orchestration tools.
Years of Experience: 12.00 Years of Experience