Role Overview
We’re looking for a hands-on Data Engineer with strong experience building reliable, real-time data pipelines using Amazon Kinesis and Databricks. You’ll work with modern AWS data technologies to develop and optimise streaming pipelines, moving high-volume data from Amazon Kinesis into the Databricks Lakehouse using Spark Structured Streaming and Delta tables.
Responsibilities
- Build and support real-time data pipelines using Amazon Kinesis Data Streams and Kinesis Data Firehose.
- Develop streaming solutions in Databricks using Spark Structured Streaming.
- Ingest and manage streaming data within Delta tables.
- Optimise pipelines for performance, latency, reliability and cost.
- Build monitoring and alerting using AWS CloudWatch.
- Work with batch data engineers to ensure streaming and batch pipelines integrate effectively.
- Develop clean, testable Python and SQL code.
- Troubleshoot pipeline and data-processing issues and contribute to continuous improvement.
Requirements
- Hands-on production experience with Amazon Kinesis Data Streams and/or Kinesis Data Firehose. Kinesis experience is essential.
- Strong Python and SQL skills.
- Experience with Apache Spark Structured Streaming.
- Hands-on experience with Databricks, ideally including Delta Lake and Unity Catalog.
- Good knowledge of AWS services, particularly IAM, S3 and CloudWatch.
- Understanding of scalable and reliable data-pipeline design.
- Strong troubleshooting and problem-solving skills.
- Experience with Kafka is beneficial
Benefits
- 18 weeks paid parental leave
- Long & short-term career break opportunities
- Structured career development program
- Local and international career opportunities.
- Certified as a Family Inclusive Workplace™
- Flexible Work Arrangements – centred around Accenture’s Truly Human ethos and our commitment to supporting the health and wellbeing of our people.
- We are proud to be in the top 3 of last year’s Diversity & Inclusion Index!