Job Summary The Senior Data Engineer is responsible for designing and developing big data applications using the latest open-source technologies.
This role will involve working in an offshore model with a focus on managed outcomes, creating data pipelines, and automating workflows.
The successful candidate will be expected to lead technical design reviews, mentor junior engineers, and participate in rotational on-call support.
Key Responsibilities - Design and develop big data applications using modern open-source technologies
- Develop logical and physical data models for big data platforms
- Automate workflows using Apache Airflow
- Create data pipelines using Apache Hive, Apache Spark, and Apache Kafka
- Maintain and enhance existing systems, providing ongoing support
- Participate in rotational on-call support
- Mentor junior engineers and lead technical design reviews
- Manage project tasks using JIRA and ensure alignment with business objectives
- Act as the point of contact for your assigned business domain and contribute to project delivery
Required Qualifications - 4+ years of experience with Google Cloud Platform (GCP)
- Experience building data pipelines in GCP, including GCP Dataproc, GCS, and BigQuery
- 10+ years of hands-on experience developing data warehouse solutions and data products
- 6+ years of experience with distributed data processing platforms such as Hadoop, Hive, Spark, and Airflow
- 5+ years of experience in schema modeling and designing for data lakes or RDBMS platforms
- Proficiency in programming languages such as Python, Java, and Scala
- Experience with scripting languages like Perl and Shell
- Strong background in Scrum/Agile methodologies
- Ability to handle multiple competing priorities with minimal supervision
- Excellent verbal and written communication skills
- Bachelor's Degree in Computer Science or a related field
Preferred Qualifications - Familiarity with Gitflow and Atlassian tools (BitBucket, JIRA, Confluence)
- Experience with continuous integration tools like Bamboo, Jenkins, or TFS
- Strong understanding of data modeling and database technologies
- Ability to optimize application development, maintenance, and performance
- Experience with large-scale data sets (multi-TB/PB scale)
Education: Bachelors Degree