Senior Data Engineer

About Us:

Perfict Global is a leading IT consulting services provider focused on providing innovative and successful business workforce solutions to Fortune 500 companies. Our trained and experienced professionals constantly strive to bring together the best technologies available to manage client's complex business and technology, participate in implementation activities and collaborate in new ways to meet client needs.

We provide excellent benefits such as Medical, Dental, Vision a fun company to work

Job description:

Senior Data Engineer: at least 8 years in data engineering with a focus on batch and streaming data

Data Engineer: between 5 and 8 years in data engineering with a focus on batch and streaming data

Both Senior and Data Engineer

Develops ELT processes and frameworks to efficiently extract, load, and transform data from various sources

Partners with DevOps to design and build scalable, automated CI/CD pipelines using Harness and GitHub

Monitors and optimizes the performance of data pipelines to ensure high data quality and availability

Responsible for the physical data structures, data integration, build, and care of solutions

Azure Data Factory

Databricks

Unity Catalog

Strong SQL writing and optimizing skills.

Strong proficiency in Python.

Expertise and professional experience with Kafka streams.

Experience creating synthetic test data for data pipelines.

Experience with workflow orchestration/scheduling solutions.

Solid understanding of modern cloud-based data architecture, ELT processes, and real-time data processing.

Able to troubleshoot and resolve issues in data pipelines.

Nice-to-have experience implementing observability solutions.