Data Engineer
Hybrid No sponsorship Contract
Overview:
- Create and optimize ETL (Extract, Transform, Load) processes using AWS services such as Glue, S3, Lambda and Data Pipelines.
- Develop scalable data architectures that support our microservice-based data ecosystem.
- Utilize Terraform for infrastructure as code (IaC) to set up and manage resources.
- Implement best practices in data modeling, ensuring efficient data structures.
- Use PySpark and Python scripting for data transformation and enrichment.
- Work with Glue crawlers to discover and catalog data sources.
- Build and maintain Lambda functions for serverless data processing.
- Leverage Glue jobs for data transformation and orchestration.
- Utilize Glue catalog for metadata management.
- Implement logging, monitoring, and alerting using AWS CloudWatch and other relevant tools.
- Work with Delta Lake and Databricks for advanced analytics and data processing.
- Query data stored in S3 data lakes using AWS Athena.
- Interface with business stakeholders to gather requirements and deliver complete reporting solutions.
- Collaborate with cross-functional teams to ensure data consistency and accuracy.
Qualifications:
- Proficiency in Python and PySpark.
- Familiarity with Terraform for infrastructure provisioning.
- Experience with Glue jobs, Lambda functions, and Glue crawlers.
- Knowledge of Delta Lake, Databricks, and S3 data lake storage.
Benefits And Compensation:
- Compensation: $100,000 - $120,000
- Competitive PTO Package
- Retirement: 401(k) plan with a match
- Medical: Individual and Family plans available