Project Description:
Migration of legacy data ingestion pipelines to a shared data management system (BDMS). The work will involve:
- Datasets will be loaded into BDMD
- New tools will be created to allow data analysts to modify the datasets
- Analysts will validate and cleanse the data which will then be delivered to AWS S3
- Data will then be enriched using technologies like Databricks, AWS Lambda & Step Functions and Jupyter Notebooks/Workflows and loaded in Delta tables.
- 5+ years of experience programming in Python.
- Solid understanding of database systems and SQL including Oracle and PostgreSQL
- A Degree in Computer Science, Engineering, Mathematics, similar field of study or equivalent work experience.
- AWS experience including services like S3, Lambda, CloudWatch
- Experience with data migrations from legacy to cloud.
- Knowledge of Spark, Databricks and IaC(Terraform/CloudFormation)
- Work directly with product owners, engineers to build solutions.
- Work with data analysts/scientists to collect insight on possible data collection efficiencies and identify automation potential for manual workflows.
- Develop infrastructure as code (IaC) and embrace DevOps principles.
- Work with AWS cloud services to implement micro services using serverless technologies.