Role Overview

Design, develop, and maintain secure and scalable data pipelines and infrastructure using AWS services such as EMR, S3, Glue, and Lambda. This role involves implementing Infrastructure as Code using Terraform, collaborating with FinOps teams for cost optimization, and ensuring robust security across a data lakehouse environment.

Responsibilities

Design, develop, and maintain data workflows and ETL processes using Amazon Glue
Write complex SQL queries to extract and manipulate data effectively
Develop Python scripts for data processing and automation tasks
Implement DevOps solutions for continuous integration and deployment of data pipelines
Work with Snowflake data warehouse for data storage and retrieval
Collaborate with cross-functional teams to understand business requirements
Troubleshoot and optimize existing data pipelines for performance and scalability

Requirements

Proven experience as a Data Engineer or Cloud Engineer working extensively with AWS services, particularly EMR, S3, Lambda, and Snowflake
Strong hands-on experience with Terraform or similar IaC tools
Knowledge of FinOps principles and cloud cost optimization
Solid understanding of cloud security best practices and identity and access management
Experience with data orchestration tools such as Apache Airflow
Proficient in scripting languages such as Python or Bash
Excellent problem-solving and communication skills

Skills

AWS
Snowflake
Python
Terraform
SQL