Azure Databricks & Generative AI Engineer

Role Overview

We are looking for an experienced Azure Databricks & Generative AI Engineer to build scalable data and AI solutions using Azure Databricks, Python, and Generative AI technologies. The ideal candidate should have strong expertise in data engineering, Databricks, Azure cloud services, and hands-on experience developing or integrating Gen AI/LLM-based solutions.

Responsibilities

• Design, develop, and maintain scalable data solutions using Azure Databricks.
• Develop data processing and transformation pipelines using PySpark and Python.
• Work with Delta Lake, Spark SQL, and Databricks Workflows.
• Build and integrate Generative AI and LLM-based applications.
• Develop AI-powered solutions using Azure OpenAI and Azure AI services.
• Implement RAG (Retrieval-Augmented Generation) solutions using enterprise data.
• Work with embeddings, vector databases, and AI Search for intelligent information retrieval.
• Integrate structured and unstructured data into Gen AI applications.
• Develop data pipelines to support AI/ML and analytics use cases.
• Optimize Databricks jobs, Spark workloads, queries, and data processing performance.
• Implement data quality, security, governance, and access controls.
• Collaborate with data scientists, software engineers, business teams, and cloud architects.
• Monitor and troubleshoot data pipelines and AI workloads.
• Follow best practices for cloud architecture, CI/CD, version control, and deployment.

Requirements

• Strong hands-on experience with Azure Databricks.
• Strong knowledge of PySpark, Python, and SQL.
• Experience with Delta Lake and Databricks Workflows.
• Good understanding of Azure cloud services.
• Hands-on knowledge of Generative AI and Large Language Models (LLMs).
• Experience with Azure OpenAI / Azure AI services.
• Understanding of RAG architecture, embeddings, prompt engineering, and vector search.
• Experience working with structured and unstructured data.
• Knowledge of data engineering and ETL/ELT concepts.
• Experience with Git and CI/CD practices.
• Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, Data Science, or a related field.

Skills

• Azure Databricks
• Python
• PySpark
• Generative AI
• Azure OpenAI