Job Summary: We are seeking a highly skilled Senior Data Engineer to join our team in Sunnyvale, CA.
The ideal candidate will have extensive experience with GCP, big data technologies, and data warehouse solutions.
You will lead the design, development, and maintenance of data pipelines and infrastructure, mentor junior engineers, and act as the primary point of contact for your assigned business domain.
Key Responsibilities: - Design and develop big data applications using open-source technologies.
- Build data pipelines using GCP Dataproc, Apache Hive, Apache Spark, Apache Kafka, and GCS.
- Automate workflows using Apache Airflow.
- Develop logical and physical data models for big data platforms.
- Provide ongoing system maintenance, enhancements, and on-call support.
- Mentor junior engineers and lead design reviews and standups.
- Groom and prioritize backlogs using JIRA.
- Interact directly with stakeholders to understand business domains and translate them into technical solutions.
- Lead project delivery activities including estimations, documentation, coding, and configuration.
- Perform defect RCA, manage release processes, and ensure compliance with development standards.
- Engage with clients to clarify requirements, present design options, and conduct product demos.
Required Qualifications: - Bachelor's degree in Computer Science or related field (or equivalent experience).
- 10+ years of experience in developing data warehouse solutions and data products.
- 6+ years with Hadoop, Hive/Spark, Airflow, or similar tools.
- 4+ years of hands-on experience with GCP (Dataproc, BigQuery, GCS).
- Strong experience in Python, Java, Scala, and scripting languages like Shell or Perl.
- Experience in building data lakes and modeling schemas.
- Practice working with large data sets (TB/PB scale).
- Experience with Agile/Scrum methodologies and test-driven development.
- Proficient in Git, BitBucket, JIRA, Jenkins/Bamboo, and Confluence.
- Strong communication, analytical, and team leadership skills.
Preferred Qualifications: - Experience with Gitflow and CI/CD tools such as Jenkins, TFS, or Bamboo.
- Background in software architecture and design principles.
- Familiarity with software development lifecycle and performance monitoring.
- Domain certifications and experience in relevant customer domains.
Education: Bachelors Degree