Data Engineer with Scala and Spark/ Pyspark

Role Overview

Capco, a Wipro company, is a global technology and management consulting firm. We are looking for a Data Engineer with 4+ years of experience to design, develop, and optimize large-scale data processing applications. This is a hybrid role based at the Capco office, working on projects that transform the financial services industry.

Responsibilities

• Design, develop, and optimize large-scale data processing applications using Scala, Apache Spark, and Java.
• Build and maintain high-performance data pipelines capable of processing 80–90 million records daily.
• Develop and integrate Native APIs and data services to support business-critical applications.
• Optimize Spark jobs, data workflows, and distributed computing processes.
• Implement best practices for data quality, monitoring, governance, and operational excellence.
• Troubleshoot and resolve performance bottlenecks across data processing and ingestion pipelines.

Requirements

• Strong hands-on experience in Scala, Apache Spark, and Java.
• Proven experience building and supporting large-scale distributed data processing systems.
• Expertise in developing and consuming Native APIs and microservices.
• Strong understanding of Spark architecture, performance tuning, and optimization techniques.
• Experience with data modeling, ETL/ELT processes, and large-scale batch and streaming data pipelines.
• Experience with SQL and relational/non-relational databases.

Nice to Have

• Experience in Financial Services, Payments, or FinTech domains.
• Exposure to cloud platforms like AWS, Azure, or GCP.
• Familiarity with Kafka, Airflow, or the Hadoop ecosystem.
• Experience with CI/CD pipelines and DevOps practices.

Skills

• Scala
• Apache Spark
• Java
• Native APIs
• SQL