Big Data Engineer

Job Title

Big Data Engineer (EL3)

Experience Required

4+ Years

Employment Type

Full-Time

Job Summary

We are looking for an experienced Big Data Engineer to design, develop, and optimize scalable big data pipelines and data validation solutions. The ideal candidate should have strong hands-on experience in Scala, PySpark, AWS, and AWS Glue, along with expertise in data validation, profiling, and modern big data technologies. The role also involves leveraging AI-assisted development tools to improve engineering productivity and software quality.

Required Skills & Experience

• 4+ years of software engineering experience developing Big Data pipelines.
• 3+ years of hands-on programming experience using Scala and PySpark.
• 3+ years of experience building cloud-based data solutions on AWS, including AWS Glue.
• 3+ years of experience working with Big Data file formats such as Parquet.
• 1+ years of experience in data validation, data profiling, or developing rule-based data validation engines.
• 1+ years of experience using AI-powered coding assistants or code-generation tools such as GitHub Copilot, OpenAI Codex, or similar tools in daily software development.

Key Responsibilities

• Design, develop, and optimize scalable, rule-based data validation engines capable of validating data across diverse data pipelines.
• Build robust, high-performance Big Data pipelines and processing platforms using Scala, PySpark, AWS, and AWS Glue.
• Develop clean, maintainable, and…

Skills

• AWS
• Scala
• Python
• Apache Spark