MANDATORY: Hands-on experience with Azure cloud and distributed processing (Spark). Experience in Azure technologies like Azure Data Factory, Azure Databricks using Python, Azure Data and Should be proficient optimizing code for performance (able to read a DAG, determine where CBO is using most resources)
JD
• Experienced SR Data Engineer: 12+ years of experience working with data infrastructure, ETL processes, and large-scale datasets.
• Proficient in SQL & Python: Strong knowledge of SQL for querying and manipulating data and experience with Python for scripting and automation.
• PowerShell experience - this is our scripting language of choice.
• Proficiency with parquet and delta formats
• Cloud & Big Data: Hands-on experience with Azure cloud and distributed processing (Spark). Experience in Azure technologies like Azure Data Factory, Azure Databricks using Python, Azure Data Lake Storage, Azure Functions, AzureDevOps for CI/CD and Logic App.
• Should be proficient optimizing code for performance (able to read a DAG, determine where CBO is using most resources)
• Strong understanding of data warehousing concepts and dimensional modeling.
• Strong understanding of software development lifecycle from planning to deployment. We follow a standard deployment process (dev, test, prod) that includes peer reviewed code
• Problem Solver & Innovator: Passionate about solving complex data challenges and improving system efficiency.
• Team Player & Communicator: Strong collaboration skills, with the ability to explain technical concepts to non-technical stakeholders.