Job Summary: The Data Engineer will be responsible for leveraging the data platform to create data products for business. This role involves the development of data products, data pipelines, data transformation, data cleansing, data normalization, deployment & support for various functions within EGIS.

Technologies: Azure Data Factory, Data Bricks, SQL, Azure DevOps, Azure Azure Devops, gitlab, Power BI

Key Responsibilities:

· Design and implement Extract, Transform, Load (ETL) processes to move data from various sources (on-premises, cloud, and third-party APIs) to Azure data platforms.

· Integrate data from diverse sources into Azure-based systems like Azure Data Lake/Azure SQL Database.

· Use Azure Data Factory or Data Bricks orchestration tools to automate and schedule data pipelines/Job workflows

· Design efficient data models (e.g., star or snowflake schema) for use in analytical applications and reporting systems.

· Optimize SQL queries and scripts for performance, ensuring low-latency and efficient data processing

· Continuously monitor the health of pipelines, jobs and infrastructure, ensuring they are running efficiently and securely

· Assist in building dashboards and reporting solutions using tools like Power BI, ensuring data is made available in an user-friendly format

· Address and resolve data issues, including failures in ETL pipelines, system performance problems, and data inconsistencies

· Ensure that all data engineering processes comply with industry security standards and best practices, including encryption, access controls, and data masking