Description: Job Location: New York City, New York - Hybrid (3 days onsite) Job Title: Azure Databricks DevOps Administrator Duration: Long term Contract / CTH Target Start Date: ASAP Experience As an Azure Databricks DevOps Administrator, you will be responsible > for managing and maintaining the Databricks platform, ensuring optimal > performance and security across workspaces, clusters, and jobs. This > role is crucial in overseeing user administration, managing Azure > storage solutions, and implementing CI/CD pipelines using Azure > DevOps. By leveraging a deep understanding of Databricks architecture > and proficiency in scripting languages, the Azure DevOps administrator > will automate tasks and enhance the efficiency of data operations. > Strong communication skills and a commitment to high-quality support > will enable effective collaboration with cross-functional teams, > directly contributing to the department's mission of delivering robust > data solutions and insights. Job Responsibilities Hands-on experience in Azure Cloud services, Networking Concepts, > Security - Cloud and on-premises system, Deployment using Azure > DevOps, Azure Cloud Monitoring and Cost controls and Terraform - > hands-on experience > CI/CD Pipeline Management: Design, implement, and manage > Continuous Integration/Continuous Deployment (CI/CD) pipelines using > Azure DevOps and GitHub. Ensure the pipelines are efficient, reliable, > and scalable. > Infrastructure as Code (IaC): Automate the provisioning and > management of infrastructure, with a focus on Azure Databricks and > Azure Data Factory in a private network environment, using tools like > Terraform and ARM templates. > Environment Management: Create and manage development, testing, > and production environments, ensuring consistency, security, and > alignment with organizational requirements. > Security: Implement security best practices throughout the CI/CD > pipeline, including secrets management, secure code scanning, and > compliance with security standards. > Monitoring & Logging: Set up and maintain monitoring and logging > for applications and infrastructure using Azure Monitor, Log > Analytics, and related tools to ensure system reliability and > performance. > Automation: Identify opportunities for automation to streamline > processes, reduce manual errors, and improve operational efficiency. > Policy Enforcement: Establish and enforce policies such as branch > policies, pull request reviews, and pipeline approvals to maintain > code quality and compliance with organizational standards. > Manage and maintain Azure Databricks Platform, workspaces, > clusters, and jobs > Oversee user administration including access controls and > permissions > Handle library installations, runtime management, and policy > enforcement > Implement/analyze cost control measures > Administer Unity Catalog for data governance and security > Collaborate with data engineers, data scientists, and analysts to > optimize and streamline data workflows, and analytical pipelines on > the Databricks platform > Manage Azure storage solutions, including Blob Storage and Data > Lake Storage > Administer Azure Key Vault for secure storage of secrets and keys > Configure and manage Azure Data Factory for data integration and > ETL processes > Implement and manage VNETs, firewalls, Azure policies and security > best practices > Set up budgets and alerts to monitor and control Azure costs and > spend > Configure alerts for proactive issue detection and resolution > Databricks Lakehouse Monitoring Minimum Qualifications Minimum of Bachelor's degree in Computer science, Information > systems, or Engineering > Experience: 7+ years of professional experience in DevOps, with a > strong focus on Azure. Knowledge of Data Platforms like Databricks, > Data Factory (preferred) > Proficiency in scripting languages such as Python/PySpark, > PowerShell, or Bash > Experience in automating administrative tasks and workflows > Knowledge of security best practices and compliance requirements > Experience with ETL processes, data pipelines, and big data > technologies > Experience with backup and restore procedures for Databricks and > Azure services > Ability to troubleshoot and resolve issues in a timely and > efficient manner > Strong verbal and written communication skills > Ability to document processes, procedures, and configurations > clearly > Commitment to providing high-quality support to internal/external > users and stakeholders > Ability to understand and address the needs of internal and > external customers > Team player with the ability to work collaboratively with > cross-functional teams > Flexibility to adapt to changing requirements and priorities > Continuous Learning: Eagerness to learn new technologies and > continuously improve skills to stay current in a rapidly evolving > field.
Date Posted: 14 April 2025
Apply for this Job