Azure Databricks & GenAI Engineer

Role Overview

We are looking for a skilled Azure Databricks & GenAI Engineer to design, develop, and maintain scalable data and AI solutions on Microsoft Azure. The ideal candidate will have strong hands-on experience with Azure Databricks, PySpark, Python, SQL, data engineering, Generative AI, and LLM-based applications. The candidate will work on building modern data pipelines, lakehouse architectures, AI/ML workflows, and GenAI solutions that leverage enterprise data.

Responsibilities

• Design and develop scalable data engineering solutions using Azure Databricks.
• Build and optimize data pipelines using PySpark, Python, SQL, and Delta Lake.
• Develop and maintain Databricks notebooks, workflows, jobs, and Delta tables.
• Implement Medallion Architecture (Bronze, Silver, Gold) for data processing and transformation.
• Develop Generative AI solutions using Large Language Models (LLMs), embeddings, and vector search.
• Build RAG (Retrieval-Augmented Generation) applications using enterprise data.
• Integrate Azure AI services and Azure OpenAI with Databricks-based solutions.
• Develop data ingestion and transformation pipelines using Azure Data Factory (ADF) and other Azure services.
• Implement data preprocessing, feature engineering, and model-ready datasets for AI/ML applications.
• Work with MLflow for experiment tracking, model management, and deployment.
• Optimize Databricks workloads, Spark jobs, SQL queries, and data pipelines for performance and cost.
• Implement data quality, governance, security, and access controls across data platforms.
• Collaborate with Data Scientists, AI Engineers, Software Developers, and business stakeholders.
• Develop CI/CD pipelines for data and AI workloads using Azure DevOps or GitHub Actions.
• Troubleshoot production data and GenAI solutions and ensure high availability and reliability.

Requirements

• Strong hands-on experience with Azure Databricks.
• Strong knowledge of Python, PySpark, and SQL.
• Experience with Delta Lake and Lakehouse Architecture.
• Practical experience with Generative AI and LLM applications.
• Knowledge of RAG, embeddings, vector databases/vector search, and prompt engineering.
• Experience with Azure OpenAI or equivalent LLM platforms.
• Experience with Azure Data Factory (ADF).
• Knowledge of MLflow and MLOps concepts.
• Understanding of data modeling, ETL/ELT, and data warehousing concepts.
• Experience with Git and CI/CD practices.
• Strong analytical and problem-solving skills.

Nice to Have

• Experience with Azure AI Search and vector search.
• Experience developing AI applications using frameworks such as LangChain or LlamaIndex.
• Knowledge of Databricks Mosaic AI and Model Serving.
• Experience with Unity Catalog and Databricks governance.
• Knowledge of REST APIs and microservices.
• Experience with Azure services such as ADLS Gen2, Azure Functions, Key Vault, and Event Hubs.
• Understanding of responsible AI, data privacy, and enterprise AI security.

Skills

• Azure Databricks
• Python
• PySpark
• Generative AI
• SQL