Cloud Platform Engineer

Role Overview

We are seeking a highly skilled Cloud Platform Engineer with strong experience in Databricks, Azure Cloud, and Snowflake to join our team for a Cloud Data Modernization project. The ideal candidate will have hands-on experience in cloud data platforms, ETL migration, data engineering, cloud modernization, CI/CD, and data platform solutions.

Responsibilities

• Contribute to the migration of ETL workloads from on-premises platforms to Azure Cloud, Databricks, and Snowflake.
• Develop and implement scalable data platform solutions using Databricks, Azure Data Factory (ADF), SHIR, Logic Apps, ADLS Gen2, Blob Storage, and Snowflake.
• Review and analyze existing on-premises ETL processes and support their modernization.
• Develop and optimize data pipelines using Apache Spark and PySpark.
• Implement DevOps practices and CI/CD pipelines using GitHub Actions.
• Collaborate with cross-functional teams to ensure seamless integration and reliable data flow.
• Tune and optimize workloads for performance, scalability, and cost efficiency.
• Implement data quality validation, monitoring, reconciliation, and observability processes.
• Support automated testing and data validation frameworks.
• Ensure data security, governance, and compliance with industry standards.
• Participate in code reviews and follow engineering best practices.
• Work effectively within Agile delivery methodologies.
• Leverage AI-assisted development tools such as GitHub Copilot, Databricks Assistant, ChatGPT, or Claude to improve engineering productivity.

Requirements

• Strong hands-on experience with Databricks.
• Experience with Azure Cloud and/or AWS cloud infrastructure.
• Hands-on experience with Snowflake.
• Strong knowledge of Apache Spark and PySpark.
• Experience with Azure Data Factory (ADF), SHIR, Logic Apps, ADLS Gen2, and Blob Storage.
• Strong experience in ETL/ELT development using on-premises databases and cloud technologies.
• Strong SQL development skills, including complex queries, stored procedures, views, and ETL transformations.
• Experience with Python or other scripting languages.
• Experience with GitHub, branching, pull requests, and GitHub Actions.
• Understanding of CI/CD, automated testing, data quality, monitoring, and observability.
• Experience with code reviews and engineering best practices.
• Experience working in Agile environments.

Skills

• Databricks
• Azure Cloud
• Apache Spark
• PySpark
• Snowflake

Nice to Have

• Experience with Delta Lake and Lakehouse architectures.
• Knowledge of Bronze/Silver/Gold (Medallion) architecture.
• Experience with Databricks and Snowflake performance optimization.
• Knowledge of data modeling and database design.
• Experience with Airflow or other orchestration frameworks.
• Knowledge of data governance and data quality best practices.
• Experience in healthcare payer data domains.
• Experience using AI/ML solutions for data engineering workflows.
• Azure or Databricks certification.