Greymatter Innovationz helps you stay digitally relevant across domains, technologies, and skillsets, every day.
We are looking for:
Lead Data Bricks Platform and Service Engineering Specialist
Location: Princeton, NJ
Duration: Long Term
Data Bricks Platform and Service Engineering Specialist
The Data Bricks Platform and Service Engineering Specialist is a key role focused on designing, managing, and optimizing Data Bricks environments and the services running on them. This specialist will bridge the gap between infrastructure management and data engineering by providing scalable, secure, and high-performing solutions on the Data Bricks platform. The role involves working closely with data engineers, platform teams, and business stakeholders to ensure seamless data workflows, robust analytics, and reliable platform operations.
This role is ideal for someone passionate about merging platform engineering with data-driven solutions and enabling teams to unlock the full potential of the Data Bricks platform. An over all experience of 10+ years and Databricks experience of at least 3-4 years is preferred.
Platform Engineering
- Design, build, and maintain the Data Bricks platform infrastructure across cloud providers (AWS or Azure), ensuring scalability, high availability, and cost efficiency.
- Develop Infrastructure as Code (IaC) solutions (e.g., Terraform, CloudFormation) to automate cluster provisioning, workspace creation, and resource management.
- Implement and manage platform security, including role-based access control (RBAC), service principals, and data encryption.
- Monitor, troubleshoot, and optimize platform performance, leveraging tools like Databricks CLI, REST APIs, and third-party observability platforms.
Service Engineering
- Collaborate with data engineering teams to deploy, monitor, and optimize data pipelines, streaming applications, and ML models on Data Bricks.
- Develop and implement strategies for integrating Delta Lake with existing data warehouses, lakes, or real-time streaming systems.
- Provide support for Spark-based applications, ensuring efficient resource utilization and troubleshooting slow-performing jobs.
- Enable seamless CI/CD workflows for notebooks, jobs, and ML models, integrating tools like Jenkins, GitHub Actions, or Azure DevOps.
Governance and Compliance
- Enforce data governance policies using Unity Catalog and other tools to ensure data quality, lineage, and regulatory compliance.
- Design and implement robust logging, monitoring, and alerting systems for platform operations.
- Work closely with stakeholders to establish governance around workspace provisioning, cluster configurations, and data access controls.
Collaboration and Leadership
- Serve as the subject matter expert for the Data Bricks platform, providing guidance to data engineers and analysts on best practices.
- Work cross-functionally with DevOps, IT, and Data Science teams to ensure platform alignment with organizational goals.
- Document platform architecture, operational procedures, and troubleshooting guides to streamline platform management and onboarding.
Required Skills and Qualifications
Technical Expertise
- Extensive experience in managing and optimizing Data Bricks environments and Spark-based workloads.
- Hands-on experience with cloud platforms (AWS, Azure, GCP) and their respective Data Bricks offerings.
- Proficiency in Infrastructure as Code (IaC) tools like Terraform, CloudFormation, or Pulumi.
- Strong programming skills in Python, Scala, and SQL.
- In-depth knowledge of Delta Lake, data partitioning, and schema evolution.
- Familiarity with MLOps tools such as MLflow for tracking experiments and managing models.
Platform and Service Management
- Expertise in CI/CD pipelines and version control systems (e.g., Git, Jenkins, Azure DevOps).
- Solid understanding of access control, data security, and compliance standards (e.g., GDPR, HIPAA).
- Experience with monitoring and observability tools (e.g., Prometheus, Datadog, CloudWatch).
Soft Skills
- Excellent problem-solving and troubleshooting skills for distributed systems and data pipelines.
- Strong communication and collaboration skills to work with diverse technical and non-technical teams.
- Ability to document technical processes and create knowledge-sharing materials.
Preferred Qualifications
- Data Bricks Certified Associate Developer or Architect certification.
- Knowledge of advanced data governance tools such as Alation or Collibra.
- Familiarity with containerization and orchestration (e.g., Docker, Kubernetes).
At Greymatter Innovationz,
We offer:
Motivating Work Environment.
Excellent Work Culture.
Help you to upgrade yourself to the next level.
And More