Ensono Technologies LLP
Associate Data Engineer
Chennai, India
Developing and supporting Azure data pipelines and medallion lakehouse processing, with a focus on ingestion, validation, incremental loading, issue investigation and reliable downstream data delivery.
- Develop and enhance a Bronze-to-Silver-to-Gold medallion lakehouse using Azure Databricks, PySpark and Delta Lake for a mid-sized US e-commerce platform.
- Build metadata-driven Azure Data Factory pipelines that ingest data from SQL Server, REST APIs and SFTP into the ADLS Gen2 Bronze layer.
- Implement JSON configuration and control-table patterns for dynamic pipeline selection, parameterised execution and reusable source onboarding.
- Apply watermark-based incremental loading and rerun-safe controls so eligible changes are processed and state updates occur only after successful execution.
- Implement PySpark transformation and validation logic for trusted Silver and Gold datasets, with Delta Lake processing and Unity Catalog registration.
- Validate runtime parameters, source paths, files, schemas and row counts before data proceeds to downstream processing.
- Integrate structured Databricks notebook results and framework logging with Azure Data Factory so orchestration status reflects processing outcomes.
- Investigate pipeline issues, implement fixes, verify downstream data readiness and document pipeline behaviour for technical handoff.