For an international project in Chennai, we are urgently looking for a Full Remote (Senior) Databricks Developer with +5 years of experience.
We are looking for a motivated contractor. Candidates need to be fluent in English.
Tasks and responsibilities:
- Collaborate with data architects and analysts to design robust data pipelines on the Databricks platform;
- Develop scalable and efficient ETL processes to ingest, transform, and store large volumes of data;
- Ensure data quality and integrity through the implementation of validation and cleansing processes. Optimize data pipelines for performance, scalability, and cost-effectiveness;
- Monitor and troubleshoot data pipeline issues to ensure seamless data flow and processing;
- Implement best practices for data storage, retrieval, and processing to enhance system performance;
- Work closely with cross-functional teams to understand data requirements and deliver solutions that meet business needs;
- Document data pipeline designs, processes, and configurations for future reference and knowledge sharing;
- Provide technical guidance and support to team members and stakeholders on Databricks-related features;
Profile:
- Bachelor or Master degree;
- +5 years of experience in Data Science roles;
- Azure Databricks for developing, managing, and optimizing big data solutions on the Azure platform;
- Programming skills in Python for writing data processing scripts and working with machine learning models;
- Advanced SQL skills for querying and manipulating data within Databricks and integrating with other Azure services;
- Azure Data Lake Storage (ADLS) for storing and accessing large volumes of structured and unstructured data and ensuring data reliability and consistency in Databricks;
- Power BI Integration for creating interactive data visualizations and dashboards;
- PowerApps Integration for building custom business applications that leverage big data insights;
- Data engineering, including ETL processes and data pipeline development;
- Azure DevOps for implementing CI/CD pipelines and managing code repositories;
- Machine Learning concepts and tools within Databricks for developing predictive models;
- Azure Synapse Analytics for integrating big data and data warehousing solutions;
- Azure Functions for creating serverless computing solutions that integrate with Databricks;
- Databricks REST API for automating tasks and integrating with other systems;
- Azure Active Directory for managing user access and security within Azure Databricks;
- Azure Blob Storage for storing and retrieving large amounts of unstructured data;
- Azure Monitor for tracking and analyzing the performance of Databricks applications;
- Have familiarity with data governance practices for ensuring compliance and data quality in big data projects;
- Fluent in English;