Title: Subcontractor
Area(s) of responsibility
Role: Debezium and Kafka Connect Application Developer
Location: DAf, Eindhoven, Netherlands
Type: 1 yr contract – Tenure as of now is one year project.
Mode: Fully Onsite
Cost to BSL: upto 55 EUR per hour
Role Overview
We are seeking an experienced Developer/Technical Application Manager to design, build, and maintain robust data ingestion pipelines. The core of this role involves leveraging Debezium for Change Data Capture (CDC) to stream data into our datalakehouse.
You will be responsible for the end-to-end lifecycle of these pipelines, operating within an AWS environment using Docker containers on ECS and ECR. This is a hybrid role requiring strong development capabilities alongside operational management (Application Management) to ensure high availability and data integrity of our data ingestion pipelines.
Key Responsibilities
Pipeline Development: Take ownership of the CDC ingestion framework utilizing Kafka connectors (Debezium, Iceberg sink, S3 sink).
Containerized Infrastructure Management: Deploy and manage Debezium and Kafka Connect workers using Docker containers orchestrating on AWS ECS (Elastic Container Service) and ECR.
Data Lake Integration: Manage data ingestion into AWS S3, utilizing Parquet and Apache Iceberg formats.
Infrastructure as Code: Use Terraform to provision and manage AWS resources supporting the data platform.
CI/CD: Build and maintain deployment pipelines using GitHub and GitHub Actions.
Operational Excellence: Monitor pipeline health, troubleshoot connectivity issues, and ensure the reliability of the Kafka ecosystem.
Optional: Support and optimize workflow orchestration using Airflow where applicable.
Candidate Profile
Must-Have Experience (Non-Negotiable)
Apache Kafka & Kafka Connect: Multiple years of hands-on experience configuring, deploying, and managing Kafka Connect clusters in a production environment.
Containerization: Extensive experience with Docker is required. You must be comfortable building images and managing container lifecycles.
AWS Compute: Proven experience running containers on AWS ECS and managing images via AWS ECR.
Key Technical Requirements
Infrastructure as Code: Strong proficiency with Terraform for provisioning AWS resources.
CI/CD: Experience creating and maintaining workflows in GitHub and GitHub Actions.
Optional: Familiarity with Apache Airflow for scheduling and monitoring is a plus but not mandatory.
Highly Desirable (Strong Plus)
Debezium: While Kafka Connect experience is a must, specific experience configuring Debezium connectors for various databases (SQL Server, PostgreSQL, etc.) is a massive advantage.
Open Table Formats: Strong knowledge of Apache Iceberg (or Hudi/Delta Lake) and storing data as Parquet on AWS S3 is significantly preferred.
Baseline Skills (Assumed)
Programming/Scripting: Proficiency in Python and Bash scripting.
Cloud Fundamentals: General knowledge of AWS core services (IAM, VPC, CloudWatch).
Optional: Basic understanding of Object-Oriented (OO) principles and software design patterns.
Tech Stack Summary
Streaming: Apache Kafka, Kafka Connect, Debezium
Compute/Containerization: AWS ECS, AWS ECR, Docker
Storage/Format: AWS S3, Apache Iceberg, Parquet
DevOps: Terraform, GitHub Actions
Languages: Python, Bash
Optional Orchestration: Apache Airflow