Role Overview
The Data Engineer will design, build, and maintain batch and streaming data pipelines, ensure data governance, and support real-time decisioning platforms within an Agile squad model.
Responsibilities
- Ingestion and transformation of data from multiple source domains (CRM feeds, third‑party data, digital signals) into the platform following design principles and governance.
- Building and maintaining batch and streaming transform pipelines and workflow orchestration.
- Data dictionary documentation and cloud object storage design for data offload scenarios.
- Collaboration with DevOps, Data Architects, and operations teams to ensure end‑to‑end pipeline reliability and observability.
- Understanding of real‑time decisioning platforms (such as Pega) and how they consume and depend on data engineering outputs.
- Working with decisioning and analytics teams to ensure accurate and timely data availability for personalization and customer engagement use cases.
- Participating in cross‑team planning and technical reviews as they relate to data dependencies for decisioning workflows.
- Delivering across multiple use cases simultaneously in an Agile squad model, managing dependencies across platforms and teams.
- Authoring and maintaining technical documentation, including solution design notes, implementation plans, and deployment runbooks.
- Actively participating in sprint ceremonies, backlog refinement, and cross‑team dependency management.
- Providing SIT, UAT, and data validation support across environments; working with QA and operations teams to validate data integrity pre‑ and post‑release.
Requirements
- Familiarity with cloud‑native data engineering patterns and object storage‑based data offload architectures (e.g. AWS S3, Azure Data Lake).
- Understanding of real‑time decisioning platforms such as Pega and their integration with data engineering layers.
- Exposure to MarTech or transformation data programs in large enterprise or telco contexts.
- Experience with workflow orchestration tools such as Apache Airflow or equivalents.
- Understanding of data governance frameworks, data lineage, and data quality processes.
- Knowledge of Agile or SAFe delivery methodology and working in cross‑functional squads.