Role: Lead Data Engineer

Experience: 8 years

Location: Hyderabad

Key Skills: ETL and Data Engineering, Azure cloud, ADF

Job Description:

We are seeking an experienced Lead Data Engineer with expertise in ETL (Extract, Transform, Load) processes, data engineering, and proficiency in Azure cloud technologies, particularly Azure Data Factory (ADF). The ideal candidate will have a minimum of 8 years of relevant experience in designing, developing, and maintaining data pipelines and data solutions.

Responsibilities:

1. Lead the design, development, implementation, and maintenance of scalable and efficient ETL processes and data pipelines.
2. Architect and build data solutions on Azure cloud platform, leveraging Azure Data Factory (ADF) and other Azure services for data ingestion, processing, and storage.
3. Collaborate with cross-functional teams including Data Scientists, Data Analysts, and Business Intelligence (BI) Developers to understand data requirements and translate them into technical solutions.
4. Design and optimize data models and schemas for performance, scalability, and reliability.
5. Develop and implement data quality checks, validation rules, and error handling mechanisms to ensure data accuracy and integrity.
6. Automate data pipeline orchestration, scheduling, and monitoring using tools like Azure Data Factory, Azure Databricks, and Azure DevOps.
7. Implement best practices for data governance, data security, and compliance with regulatory standards.
8. Mentor junior data engineers and provide technical guidance and support as needed.
9. Stay updated on emerging trends and technologies in data engineering, cloud computing, and big data analytics.
10. Participate in the evaluation and selection of new tools, technologies, and platforms to enhance the data engineering capabilities of the organization.