Senior Data Engineer

Sunnyvale, California

Compunnel
Job Expired - Click here to search for similar jobs
Job Summary

The Senior Data Engineer is responsible for designing and developing big data applications using the latest open-source technologies.

This role will involve working in an offshore model with a focus on managed outcomes, creating data pipelines, and automating workflows.

The successful candidate will be expected to lead technical design reviews, mentor junior engineers, and participate in rotational on-call support.

Key Responsibilities
  • Design and develop big data applications using modern open-source technologies
  • Develop logical and physical data models for big data platforms
  • Automate workflows using Apache Airflow
  • Create data pipelines using Apache Hive, Apache Spark, and Apache Kafka
  • Maintain and enhance existing systems, providing ongoing support
  • Participate in rotational on-call support
  • Mentor junior engineers and lead technical design reviews
  • Manage project tasks using JIRA and ensure alignment with business objectives
  • Act as the point of contact for your assigned business domain and contribute to project delivery
Required Qualifications
  • 4+ years of experience with Google Cloud Platform (GCP)
  • Experience building data pipelines in GCP, including GCP Dataproc, GCS, and BigQuery
  • 10+ years of hands-on experience developing data warehouse solutions and data products
  • 6+ years of experience with distributed data processing platforms such as Hadoop, Hive, Spark, and Airflow
  • 5+ years of experience in schema modeling and designing for data lakes or RDBMS platforms
  • Proficiency in programming languages such as Python, Java, and Scala
  • Experience with scripting languages like Perl and Shell
  • Strong background in Scrum/Agile methodologies
  • Ability to handle multiple competing priorities with minimal supervision
  • Excellent verbal and written communication skills
  • Bachelor's Degree in Computer Science or a related field
Preferred Qualifications
  • Familiarity with Gitflow and Atlassian tools (BitBucket, JIRA, Confluence)
  • Experience with continuous integration tools like Bamboo, Jenkins, or TFS
  • Strong understanding of data modeling and database technologies
  • Ability to optimize application development, maintenance, and performance
  • Experience with large-scale data sets (multi-TB/PB scale)

Education: Bachelors Degree
Date Posted: 03 May 2025
Job Expired - Click here to search for similar jobs