Azure Databricks Architect

Hello ,

Our client is looking Databricks Architect for Long term project in Minneapolis, MN (Hybrid) below is the detailed requirements.

Job Title : Azure Databricks Architect

Location : Minneapolis, MN (Hybrid)

Duration : Long term

Responsibilities:

• Bachelor's degree in Computer science or equivalent, with minimum 14+ Years of relevant experience.
• Candidate to build a Delta Lake. An understanding on Medallion architecture is important.
• Designed Batch and Real time data ingestion end to end.
• Data Lineage using Data Catalog - maintained that and used the tool to discover data sources and targets;
• Designed Data Quality Strategies - identify CDE -critical data elements, assigning data stewards from business side and coded the DQ metrics generation in curated zone/silver layer.
• Designed CDC/incremental loads, full load strategies - using hash keys, key columns and merge designs.
• Designed Data Security - PII/NPI, RBAC- role based access control.
• Azure Data factory pipeline creation Azure DevOps
• Designed using Azure Data factory pipeline and Databricks - writing out to azure blob and then finally to snowflake as consumption were from Power BI /snowflake.
• Initially worked with client team and did extensive data analysis to create the model, data delivery mechanism, cloud storage, retention requirement, security, RBAC role based access control etc
• Streaming datasets/file- were captured using AUTOLOADER - delta live tables into raw and designed to be updated into silver, gold.
• Provide guidance on Azure cost management strategies, including resource tagging, budgeting, and monitoring.
• Conduct Azure security assessments, vulnerability scanning, and threat modeling to enhance cloud security.
• Discussing potential projects and their parameters with clients, executives, and software developers.
• Planning out the blueprints for software projects, including defining the scope, allocating resources, setting deadlines, laying out communication strategies, and indicating tests and maintenance.

Job Description:

• Assembling and leading the project team.
• Participating in and supervising each stage of the project.
• Ensuring each project stays on schedule and adheres to the deadlines.
• Determining and overseeing consistent testing, evaluation, and troubleshooting of all products in all stages of completion.
• Tracking milestones, deliverables, and change requests.
• Serving as a liaison to communicate information regarding changes, milestones reached, and other pertinent information.
• Preparing project proposals and discussing potential projects with clients and stakeholders
• Facilitating project initiation by defining project scope and requirements, and preparing the necessary documents and requirements
• Developing project plans and timelines to ensure the timely submission of project deliverables
• Managing project budgets and resources to ensure the timely completion of milestones
• Identifying and managing project risks
• Facilitating team meetings and collaboration
• Liaising for changes and negotiations with relevant stakeholders
• Ensuring software quality standards are met and requirements are submitted within budget and on time