Big Data Engineer(EL3)

Job Title: Big Data Engineer (EL3)

Experience: 4+ Years

Employment Type: Full-Time

Job Summary

We are looking for a skilled Big Data Engineer with 4+ years of experience in designing, developing, and optimizing scalable big data solutions. The ideal candidate should have strong expertise in Scala, PySpark, AWS, and AWS Glue, with hands-on experience building high-performance data pipelines and data validation frameworks. The candidate will work closely with cross-functional teams to deliver scalable, high-quality data engineering solutions while leveraging AI-powered development tools to improve productivity.

Key Responsibilities

• Design, develop, and optimize highly scalable, rule-based data validation engines capable of validating data across multiple data pipelines.
• Build and maintain high-performance big data pipelines using Scala, PySpark, AWS, and AWS Glue.
• Develop clean, efficient, and well-tested code while utilizing optimized data formats such as Parquet for high-performance storage and retrieval.
• Perform data validation, profiling, and quality assessments to ensure consistency and accuracy across enterprise data systems.
• Leverage enterprise-approved AI coding assistants and code-generation tools to improve software development efficiency and automate repetitive tasks.
• Collaborate with business stakeholders and cross-functional engineering teams to translate business requirements into scalable technical solutions.
• Evaluate emerging big data…

Skills

• AWS
• Scala
• Apache Spark
• SQL
• Git