Azure Data Engineer

Job Description

We are looking for a Senior/Principal Azure Data Engineer to design, build, and maintain scalable, reliable, and cost-effective data pipelines and data products using Microsoft Azure technologies.

Key Responsibilities

• Design, develop, and maintain end-to-end data pipelines and ETL/ELT workflows using Azure Data Factory, Azure Synapse Analytics, Azure Databricks, and Azure Data Lake Storage Gen2.
• Work with storage solutions such as Azure Blob Storage for data ingestion, processing, and storage.
• Build and optimize big data processing pipelines using Apache Spark (PySpark preferred).
• Ensure data quality, validation, governance, and reliability across data pipelines.
• Collaborate with data scientists, analysts, architects, and stakeholders to deliver data solutions.
• Optimize performance, scalability, and cost efficiency of data workloads in Azure.
• Implement CI/CD pipelines, version control, testing, and deployment automation for data solutions.
• Monitor, troubleshoot, and resolve data pipeline issues proactively.
• Maintain technical documentation and best practices for data engineering standards.
• Stay updated with emerging data engineering and cloud technologies.

Requirements

• Bachelor’s degree in Computer Science, Engineering, or related field (or equivalent experience).
• Strong experience in Azure Data Engineering and ETL development.
• Proficiency in SQL using systems like SQL Server and PostgreSQL.
• Strong understanding of software engineering principles (CI/CD, version control, testing).
• Strong programming skills in Python (preferred) or Scala/Java/C#.
• Experience working with file formats like Apache Parquet, Delta Lake, and Avro.
• Strong problem-solving, analytical, and communication skills.

Preferred Skills

• Technical leadership and stakeholder management
• Experience working in consulting or client-facing environments
• Strong learning agility and adaptability in cloud ecosystems
• Experience building production-grade Spark applications using PySpark