Project Scope: Multi-cloud, retail data platform on Azure, Data warehouse is Snowflake. Also have GCP / Big Query. Long term plan will be to move some of the Azure apps / pipelines to GCP where a separate planning team that did a mapping exercise - looking for replacements to tools like Databricks (what is the equivalent of what's on Azure to GCP).
Despite the long term goal, THESE teams are doing separate initiatives on the Azure platform - Supply Chain utilizing ML models help optimize demand forecasting.
- Demand forecasting for what is sold in the stores
- The ML models handle a lot of the recommendation - they QA the data assets that go into the models & put them into production
- Enhance pipelines, add extra features etc.
- Need to also consider how they're going to ship the products
- Data modeling
- Building new pipelines
FOCUSED ON SUPPORTING THE ML MODELS, NOT BUIDING THEM
IF they have experience with a migration to GCP, there will be a higher chance they convert and get moved to that project.
Development Methodology: Agile - 2 week sprints
High-Level Individual Duties:
- Expectations are for people to run with a solution from end to end - Architect is tied to every project
- Build new pipelines, automation etc.
Must-Haves (Concepts & Tools):
- Solid Data Engineering fundamentals
- Python & SQL
- Cloud Services / Concepts (Azure preferred, GCP very nice to have)
- Airflow for orchestration (not a deal breaker to miss on this piece)
- AKS for infrastructure - should have familiarity with containers
- Should also have strong fundamentals with CI/CD pipelines
- Automation experience
Nice-to-Haves:
- GCP / Big Query
- ML Experience - again, not building models, just folding them into data pipelines.
Required Skills :
• Solid Data Engineering fundamentals o Python & SQL o Cloud Services / Concepts (Azure preferred, GCP very nice to have)
• Airflow for orchestration (not a deal breaker to miss on this piece)
• AKS for infrastructure - should have familiarity with containers o Should also have strong fundamentals with CI/CD pipelines
• Automation experience
Background Check :Yes
Drug Screen :Yes
Notes :
Selling points for candidate :
Project Verification Info :
Candidate must be your W2 Employee :Yes
Exclusive to Apex :No
Face to face interview required :No
Candidate must be local :No
Candidate must be authorized to work without sponsorship No
Interview times set No
Type of project :Development/Engineering
Master Job Title :Big Data: Other
Branch Code :Providence