Azure Databricks Engineer

PositionID: Client: Gainwell Technologies Position: Azure Databricks Engineer Responsibilities: Data pipeline Development: Design, develop, and maintain scalable data pipelines and ETL processes using Azure Databricks, Azure Data Factory, and other Azure services. Data Processing: Implement and optimize Spark jobs, data transformations, and data processing workflows in Databricks. Integration: Integrate data from various sources, such as databases, APIs, and streaming data, into the Databricks environment, often using tools like Azure Data Factory or Databricks Workflows. Performance Optimization: Enhance the performance of data processing tasks by optimizing Spark jobs, managing cluster resources, and implementing best practices for data storage and retrieval. Collaboration: Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and ensure the solutions meet business needs. Data Governance: Implement data governance policies and ensure data security and compliance with relevant regulations. CI/CD Integration: Utilize Azure DevOps and CI/CD best practices to automate the deployment and management of data pipelines and infrastructure.