Avanade logo for Databricks Data Engineer Consultant job on DevsUnite

Databricks Data Engineer Consultant

Avanade

Actively HiringFull-time
Apply
Location

Melbourne, Victoria, Australia

Experience

3+ years

Role

Databricks Data Engineer Consultant

Salary

Competitive

Share this job

Know someone who'd be a great fit?

Job Description

Role Overview

We are seeking experienced Databricks Data Engineers to join our Data and AI team. You will play a pivotal role in designing, developing, and maintaining scalable data pipelines and transformation workflows on Azure Databricks, enabling advanced analytics and data engineering initiatives.

Responsibilities

  • Design, develop, and optimize scalable data ingestion pipelines using Azure Data Factory, Databricks, and Apache Spark.
  • Build and maintain Databricks notebooks using PySpark for data preparation, transformation, and enrichment to support data science use cases.
  • Collaborate closely with data scientists to understand data requirements and ensure timely delivery of clean, structured datasets.
  • Implement data quality checks, validation rules, and monitoring mechanisms.
  • Integrate data from various sources including structured, semi-structured, and unstructured formats.
  • Ensure data security, governance, and compliance with banking regulations.
  • Optimize Spark jobs for performance and cost-efficiency in Azure.
  • Participate in code reviews, design discussions, and agile ceremonies.
  • Document data pipelines, workflows, and technical decisions.

Requirements

  • Experience in data engineering, with at least 3+ years hands-on experience in Azure Databricks.
  • Strong proficiency in Apache Spark, especially PySpark for distributed data processing.
  • Experience with Azure Data Factory, Azure Synapse, Azure Blob Storage, and Delta Lake.
  • Solid understanding of ETL/ELT processes, data modeling, and data warehousing concepts.
  • Proficiency in Python and/or Scala for data processing.
  • Experience with CI/CD pipelines using tools like Azure DevOps, Git, and Terraform (preferred).
  • Familiarity with SQL, NoSQL, and data lake architectures.
  • Knowledge of data governance, security, and compliance in financial services.

Required Skills

Azure DatabricksApache SparkPySparkAzure Data FactoryPython

Ready to apply for Databricks Data Engineer Consultant?

See how you stack up at Avanade before you apply, or go straight to the application.

Apply Anyway