Role Overview

We are seeking a skilled Data Engineer with AI/ML exposure responsible for designing, building, and maintaining scalable data pipelines and supporting data-driven applications, including AI/ML use cases. The ideal candidate should have strong expertise in data engineering tools along with working knowledge of machine learning workflows and cloud-based data platforms.

Responsibilities

Design, develop, and maintain scalable ETL/ELT pipelines
Build and optimize data architectures, data lakes, and data warehouses
Ensure data quality, integrity, and security across systems
Work with structured and unstructured data from various sources (Big Data & Cloud)
Develop solutions using tools such as Azure Data Factory / AWS Glue / GCP Dataflow
Work with big data technologies like Spark, Hadoop, or Databricks
Manage data storage solutions including S3, ADLS, BigQuery, Snowflake, or Redshift
Support machine learning pipelines and data preparation for ML models
Collaborate with Data Scientists to enable feature engineering and model deployment
Work on AI-enabled data solutions (e.g., NLP, recommendation systems, prediction models)
Basic understanding of ML frameworks (Scikit-learn, TensorFlow, or PyTorch is a plus)
Design and implement data models (dimensional & normalized)
Optimize queries and pipelines for efficiency and cost
Work closely with business teams, analysts, and ML engineers
Implement data governance, lineage, and compliance standards
Document workflows, pipelines, and architectures

Requirements

Core Data Engineering

Strong in SQL, Python
Experience with ETL tools and pipeline orchestration (Airflow, ADF, etc.)
Hands-on with data warehousing concepts

Big Data Technologies

Apache Spark / PySpark
Hadoop ecosystem (optional but preferred)

Cloud Platforms (any one required)

Azure / AWS / GCP hands-on experience
Familiarity with cloud-native data services

AI/ML Exposure

Experience working with data for ML models
Knowledge of ML lifecycle and data preparation
Exposure to MLOps concepts (bonus)

Preferred Qualifications

Experience with Databricks / Snowflake
Knowledge of API-based data ingestion
Familiarity with CI/CD pipelines
Exposure to real-time streaming (Kafka, Event Hub, etc.)
Understanding of Generative AI or LLM integrations (added advantage)