Confluent Kafka Data Engineer

Role Overview

We are seeking an experienced Data Engineer to design, develop, and maintain robust, scalable data pipelines, focusing heavily on Azure services and real-time data ingestion using Confluent Kafka.

Responsibilities

• Design, develop, and maintain robust, scalable data pipelines using Azure Data Factory (ADF) and Databricks.
• Implement batch and real-time data ingestion frameworks leveraging Confluent Kafka.
• Build and optimize data models, transformations, and ETL/ELT pipelines using SQL and Spark.
• Develop and manage data storage solutions in Azure Data Lake Storage (ADLS).
• Ensure high data quality, integrity, and governance standards.
• Support data integration across systems including core banking and CRM.
• Optimize performance of large-scale datasets.
• Collaborate with stakeholders to translate requirements into solutions.
• Ensure compliance with banking regulations and data privacy.
• Troubleshoot production issues.

Requirements

• Strong expertise in SQL (complex queries, tuning, modeling).
• Hands-on experience with Databricks, Spark, PySpark.
• Experience with Confluent Kafka and event-driven architecture.
• Experience with Azure Data Factory (ADF).
• Experience with Azure Data Lake Storage (ADLS).
• Understanding of data warehousing concepts.
• Familiarity with CI/CD and DevOps for data platforms.
• Domain Knowledge (Preferred): Experience in banking or financial services, understanding of core banking systems, knowledge of payments, lending, wealth management, and awareness of regulatory frameworks (KYC, AML).
• Good to Have: Experience with Python or Scala, exposure to data governance tools, knowledge of APIs and microservices, experience with BI tools like Power BI.
• Soft Skills: Strong analytical and problem-solving skills, excellent communication and stakeholder engagement, ability to work in fast-paced environments, strong ownership and collaboration mindset.
• Qualifications: Bachelor’s or Master’s degree in relevant field. Azure/Data engineering certifications preferred.