Role Overview
We are seeking a Senior Data Engineer to design, build, and optimize next-generation data architecture on Google Cloud Platform, including the BigQuery warehouse, while supporting data engineering responsibilities.
Responsibilities
- Design and implement complex ETL/ELT pipelines from disparate sources into a centralized data lake or warehouse.
- Manage large-scale Apache Spark clusters and perform petabyte-scale transformations using Python or Scala.
- Write and optimize complex SQL queries for data modeling and BigQuery performance.
- Develop real-time processing using Pub/Sub and Dataflow.
- Manage workflow automation using Cloud Composer and architect storage and databases across GCP.
Requirements
- Mastery of Python or Scala.
- Proven experience tuning and debugging Spark jobs at scale.
- Deep hands-on experience with the full Google Cloud data suite, including BigQuery, Dataflow, Dataproc, and Pub/Sub.
- Ability to perform complex window functions, CTEs, and query optimization.