Specialist Software Engineer - Bigdata

Role Overview

As a Specialist Software Engineer - Bigdata, you will work as an individual contributor on a Big Data Migration Project focused on Lakehouse migration. You will be responsible for migrating and testing applications on Snowflake, Snowpark, Apache Spark, and Hive within a new retail banking data Lakehouse environment.

Responsibilities

• Perform tasks related to migrating and testing applications on Snowflake, Snowpark, Apache Spark, and Hive.
• Perform extensive testing of migrated code, debugging, and fixing issues.
• Manage DevOps as required and defined on the project.
• Automate processes to improve testing team efficiency.
• Coordinate with geographically diverse teams and participate in Agile ceremonies.
• Conduct bug-free release validations and produce metrics, tests, and defect reports.
• Act as a SPOC for the team, providing status updates to management.

Requirements

• 3 to 7 years of experience on big data platforms and at least 2 years of experience implementing DWH on Snowflake.
• Proven experience with cloud platforms, preferably Azure.
• Good understanding of distributed computing frameworks like Apache Spark and Hadoop.
• High proficiency in SQL and at least one of the following: Scala or Python.
• Experience migrating data from on-premise databases to Snowflake.
• Expertise in building robust ELT/ETL processes and performance tuning of data pipelines.
• Strong knowledge of integration concepts, design best practices, and data modeling.

Skills

• Snowflake
• Apache Spark
• Python
• SQL
• Azure