Title: Technical Lead-Data Engg
Area(s) of responsibility
Sr Data Engineer:
Essential Duties and Responsibilities:
• This section contains a list of five to eight primary responsibilities of this role that account for 5% or more of the work. The incumbent will perform other duties assigned.
• Development of new ETL/data transformation jobs, using PySpark or Python in AWS.
• Enhancement and support on existing ETL/data transformation jobs.
• Can explain technical solutions and resolutions with internal customers and communicate feedback to the ETL team.
• Perform technical code reviews for peers moving code into production.
• Perform and review integration testing before production migrations.
• Provide high level of technical support and perform root cause analysis for problems experienced within area of functional responsibility.
• Can document technical specs from business communications.
Qualifications:
To perform this job successfully, an individual must be able to perform each essential duty satisfactorily. List knowledge, skills, and/or abilities required. Reasonable accommodations may be made to enable individuals with disabilities to perform essential functions.
• 6+ years of ETL experience.
• Experience with core Python programming for data transformation.
• Intermediate-level Python and PySpark skills. Can write, read, understand and debug existing code Python and ySpark code from scratch.
• Strong knowledge of SQL fundamentals and snowflake experience, understanding of subqueries, can tune queries with execution hints to improve performance.
• Able to write SQL code sufficient for most business requirements for pulling data from sources, applying rules to the data, and stocking target data
• Proven track record in troubleshooting ETL jobs and addressing production issues like performance tuning, reject handling, and ad-hoc reloads.
• Proficient in developing optimization strategies for ETL processes.
• Basic AWS technical support skills. Has ability to log in, find existing jobs and check run status and logs
• Will run and monitor jobs running via Control-M
• Can create clear and concise documentation and communications.
• Can document technical specs from business communications.