Title:  Technical Lead-Data Engg

Area(s) of responsibility

JD for Azure Data Bricks:

Core responsibilities

Design and development:

Architect and build scalable, robust data solutions on the Azure Databricks platform. 

 Data pipelines:

Develop and maintain data pipelines, handling both batch and streaming data for various file formats like CSV, JSON, and Parquet. 

 ETL/ELT processes:

Implement and optimize ETL/ELT processes to transform raw data into usable formats for analytics and reporting. 

 Collaboration:

Work closely with data engineers, data scientists, and business analysts to meet data requirements and support machine learning workflows. 

 Performance optimization:

Tune Databricks clusters and jobs for performance, scalability, and cost efficiency. 

 Integration:

Integrate Databricks with other Azure services such as Azure Data Lake, Synapse, Key Vault, and SQL Database. 

 Maintenance and support:

Monitor and troubleshoot data pipelines, resolve issues, and perform code reviews. 

 Security and governance:

Implement security best practices, access controls, and data governance policies. 

 Required skills and experience

Technical proficiency:

• Strong expertise in Azure Databricks and its components. 
• Proficiency in programming languages like PySpark and Python. 
• Strong SQL skills. 

 Azure services: Experience with other Azure services like Azure Data Factory, Azure Data Lake Storage (ADLS), and Azure SQL Database