Freelance Big Data Engineer

Company Description

ThreatXIntel is a startup cyber security company focused on protecting businesses and organizations from cyber threats. Our team offers services such as cloud security, web and mobile security testing, cloud security assessment, and DevSecOps. We provide customized, affordable solutions to meet the specific needs of our clients, ensuring they have access to high-quality cyber security services.

About the Role

We are looking for a highly skilled Big Data Engineer with hands-on experience in Scala, Apache Druid, Automic Workload Automation, and Google Cloud Platform (GCP) services including Dataproc. The role involves building and maintaining data pipelines, integrating data sources, optimizing query performance, and managing large-scale data workflows in a cloud-native environment.

Key Responsibilities

• Design, develop, and optimize ETL pipelines using Scala on GCP Dataproc
• Integrate and maintain real-time and batch data processing with Apache Druid
• Implement and automate workflows using Automic (UC4)
• Collaborate with Data Analysts and Product Managers to deliver clean and reliable data
• Optimize performance of distributed data processing jobs and Druid queries
• Ensure high availability and scalability of the data infrastructure on GCP
  

Required Skills & Experience

• Strong hands-on programming experience with Scala
• Deep understanding of Apache Druid architecture, ingestion, and querying
• Proficiency in GCP services, especially Dataproc, Cloud Storage, and BigQuery
• Experience with Automic (UC4) for job scheduling and automation
• Strong understanding of distributed data processing and performance tuning
• Experience with large-scale data systems and cloud-native development
  

Nice to Have

• Familiarity with additional GCP services (e.g., Pub/Sub, Dataflow)
• Understanding of data modeling and dashboarding best practices
• Knowledge of Spark and Hive within Dataproc
  

Why Join Us?

• Work on a high-impact, real-time analytics project
• Flexible remote working hours
• Collaborative and innovative team culture