Data Engineer

Role Overview

At Quest Global, we are an engineering services provider with over 25 years of experience, driven by the desire to make a positive difference. We bring together technologies and industries to solve complex problems through a diverse and empowered workforce.

Responsibilities

• Cooperate with both Quest Global and customer teams within a collaborative development framework.
• Write efficient and reusable code in Scala and/or Python (PySpark).
• Process large-scale structured and unstructured datasets.
• Perform data transformations, aggregations, and joins across multiple sources.
• Optimize Spark jobs for performance and resource utilization.
• Work with distributed storage systems like S3 / HDFS.
• Debug and troubleshoot production data issues.
• Collaborate with data engineers, analysts, and stakeholders.
• Ensure data quality, consistency, and reliability.
• Collaborate with cross-functional teams to analyze, design, and implement new applications.
• Ensure optimal performance, quality, and responsiveness of the application/services.

Requirements

• Strong hands-on experience with Apache Spark.
• Proficiency in Scala and/or Python (PySpark).
• Good understanding of Spark internals (RDD, DataFrame, Dataset APIs).
• Experience with data formats like Parquet, Avro, JSON.
• Familiarity with distributed systems and big data concepts.
• Strong SQL skills.
• Experience with cloud platforms (AWS preferred – S3, EMR, Glue, Kinesis, Firehose, Hive).
• Knowledge of performance tuning and optimization techniques.
• Experience with CI/CD pipelines.
• Exposure to streaming frameworks (Spark Streaming).
• Familiarity with workflow orchestration tools like Apache Airflow.
• Mandatory requirement to work from the Customer Office in Pune location.

Skills

• Apache Spark
• Python
• Scala
• AWS
• SQL