Data Engineer Databricks

Databricks, ETL pipelines, Docker for containerization, utilizing REST API in Python, Python Experience level: Mid-senior Experience required: 10 Years Education level: Bachelors degree Job function: Information Technology Industry: Information Technology and Services Pay rate : View hourly payrate Total position: 1 Relocation assistance: No Visa sponsorship eligibility: No Job Summary Strong experience in Databricks Expertise implementing batch and real time data process solutions using Azure Data Lake storage, Azure Data Factor and Databricks. Experience in built ETL pipelines for ingesting, transforming and loading data from multiple sources into Cloud data ware houses Proficient in Docker for containerization, utilizing REST API in Python for seamless system integration, and applying containerization concepts to improve deployment efficiency and scalability. Experience in data extraction, data acquisition, transformation, data manipulation, performance tuning and data analysis. Experience in Python libraries to build efficient data processing workflows and streamline ETL operations across large data sets and similar distributed systems. Expertise in automating data quality checks, reducing data errors by 40% and ensuring more reliable reporting and analytics with data marts. Expertise in data orchestration and automation tools such as Apache Airflow, Python, and PySpark, supporting end- to-end ETL workflows. Experience in deployment activities.