Role Overview
Develop and enhance scalable data-processing capabilities for critical client and regulatory reporting. Build reliable pipelines, integrate new data sources, and improve data quality and performance across the monitoring platform.
Responsibilities
- Design and develop scalable data-processing pipelines using Python, PySpark, and Databricks.
- Build complex SQL and data-processing solutions on Snowflake.
- Develop ingestion and transformation capabilities for relational and NoSQL databases.
- Integrate data from MongoDB, KDB, and application sources.
- Expand monitoring coverage and implement data-quality controls.
- Troubleshoot processing issues and optimize pipeline performance.
Requirements
- 5+ years of hands-on software engineering or data engineering experience.
- Strong professional development with Python.
- Hands-on experience building distributed data-processing solutions with PySpark.
- Experience developing data pipelines using Databricks.
- Advanced SQL development skills.
- Experience with Snowflake or comparable cloud data platforms.