Role Overview

This role focuses on designing, developing, and implementing robust data quality monitoring pipelines and data feeds, primarily utilizing AWS services and integrating with Collibra and IDMC systems.

Responsibilities

Design, develop, and implement a scalable data quality monitoring pipeline within AWS Infrastructure
Design, develop, and implement data feeds for systems that intend to consume data from Collibra
Centralize job run metadata from IDMC and Collibra to feed into an operational dashboard
Streamline regression testing for IDMC and Collibra jobs

Requirements

Strong understanding of Agile methodologies and project management tools like Jira and Confluence.
Experience in building CI/CD pipelines and automation using Jenkins & Terraform
Experience in code version control and best practices
Hands-on experience with AWS Lambda, Glue, S3, IAM, SNS, Step Functions, SQS, and DataSync
Experience in calling REST APIs
Intermediate development experience with Python

Nice to Have

Knowledge of IDMC REST APIs
Knowledge of Collibra REST APIs
Working experience with IDMC Cloud Data Governance and Catalog, Cloud Data Profiling, Metadata Command Center, and Cloud Data Quality
Working experience with Collibra
Working experience with Tableau or Amazon Quick Sight