Role Overview
As a Data Engineer, you will design, build, and maintain data infrastructure for AI and analytics, ensuring seamless integration and operation.
Responsibilities
- Design, implement, and maintain robust, scalable, and fault-tolerant ETL/ELT processes leveraging AWS services (e.g., Redshift, RDS, Glue, EMR)
- Develop and optimize data workflows using Python, PySpark, and SQL, ensuring top performance
- Work closely with data scientists, analysts, and stakeholders to gather data requirements while delivering effective solutions
- Ensure adherence to best practices related to data governance, quality, and security across all data pipelines
- Automate and optimize data processing tasks to improve reliability and reduce manual intervention
- Drive adoption of emerging big data technologies, cloud engineering, and automation practices
- Provide technical mentoring and guidance to junior team members as required
- Foster a culture of teamwork, openness, and innovation within and across teams
Requirements
- 3+ years of experience in developing and managing large-scale data infrastructure and AI applications
- Proficiency in designing and implementing scalable, fault-tolerant data infrastructure on AWS using services such as AWS Redshift, RDS, Glue, EMR, and other related tools
- Advanced skills in Python, PySpark, and SQL for efficient data manipulation and processing along with experience with data modelling and database design (SQL and NoSQL)
- Familiarity with big data technologies (e.g., Apache Spark, Hadoop) and real-time data processing solutions
- Understanding of data governance, data quality, and security best practices
- Knowledge of data streaming technologies such as Kafka, AWS Kinesis, or equivalent
- Ability to work directly with data scientists, analysts, and cross-functional teams to understand business requirements and translate them into technical solutions
- Excellent communication, leadership, and problem-solving skills
Education
- Bachelor's degree in Computer Science, Software Engineering, Data Science, or a related field
- Master’s degree in a related field, contributing added knowledge will be preferred
- AWS-related certifications (e.g., AWS Certified Data Analytics) or equivalent cloud platform credentials are highly desirable.