We have an exciting opportunity for a Data Platform Engineer with the top leading multimedia and creative software company in the world using cutting-edge tools and technologies, including industry-leading AI and creative software, empower millions of users to achieve their creative and business goals.
Responsibilities:- Setup and maintain production scale Databricks environment on public cloud such as Microsoft Azure and AWS
- Setup and maintain production scale data storage such as ADLS and S3 for multiple tenant teams using our Data Platform
- Setup and maintain production scale micro services to support the daily operation of our data platform. Services include job scheduling, security, financial, and administrative services, etc.
- Provide triage and guidance to the team on various support issues raised by our tenants
- Develop tools and automation solutions for configuration management, service deployments, monitoring, and alerting to assist with daily RTB operations
- Assure security and privacy compliance and implement Adobe Security & Compliance solutions to lock down data stored in our data lake
- Explore GenAI technologies and find opportunity to integrate them with our data platform, providing technical enhancement or improve user experience in the end
- Work with various 3rd party vendors for troubleshooting, proof of concept, and other collaborative projects to enhance our product.
Skills and Experience:- BS in Computer Science, Computer Engineering, or similar.
- Cloud Infrastructure Administration and Automation:
- AWS
- Azure
Proficient with following storage technologies:- ADLS Gen2
- S3
- Minio
- MongoDB
- Vector Databases
Setup and operation of:- Databricks
- Airflow
- Azure Kubernetes Service or Elastic Kubernetes Service
Ability to setup monitoring and alerting with:- DBX System Tables
- Prometheus
- Splunk
- ELK
Familiar with how to troubleshoot and maintain:- Servers with Linux system
- Kubernetes environment
Knowledge to operate with: Compensation: