Role Overview
Design, develop, and maintain scalable data pipelines to collect, process, transform, and manage large volumes of structured and unstructured data. Collaborate with data analysts, data scientists, and business teams to support business intelligence and reporting requirements while ensuring data quality, security, and governance.
Responsibilities
- Design, develop, and maintain scalable data pipelines.
- Collect, process, transform, and manage large volumes of structured and unstructured data.
- Develop and maintain ETL/ELT workflows.
- Build and optimize data warehouses and data lakes.
- Integrate data from multiple sources and systems.
- Perform data cleaning, validation, and transformation.
- Monitor data pipeline performance and troubleshoot issues.
- Ensure data quality, accuracy, consistency, and reliability.
- Work with data analysts, data scientists, and business teams.
- Develop and optimize SQL queries for data processing.
- Implement data security and governance practices.
- Automate data integration and processing tasks.
- Maintain technical documentation for data workflows.
- Support business intelligence and reporting requirements.
Requirements
- Strong knowledge of SQL and database management.
- Programming knowledge in Python, Java, or Scala.
- Understanding of ETL and ELT processes.
- Knowledge of Data Warehousing concepts.
- Experience with databases such as MySQL, PostgreSQL, and SQL Server.
- Knowledge of Big Data technologies such as Apache Spark and Hadoop.
- Familiarity with Apache Kafka and Apache Airflow.
- Understanding of data modeling and database optimization.
- Knowledge of cloud platforms such as AWS, Microsoft Azure, or Google Cloud.
- Understanding of data lakes and data lakehouse architecture.
- Knowledge of Git and version control systems.
- Basic understanding of Linux and shell scripting.
- Strong analytical and problem-solving skills.
- Good communication and teamwork skills.
- Attention to detail.
- Ability to work with large and complex datasets.
- Critical thinking and logical reasoning.
- Time management and organizational skills.
- Ability to learn new technologies quickly.
Nice to Have
- Python / Java / Scala.
- SQL / MySQL / PostgreSQL.
- Apache Spark / PySpark.
- Hadoop.
- Apache Kafka.
- Apache Airflow.
- ETL / ELT Tools.
- AWS / Microsoft Azure / Google Cloud.
- Data Warehousing.
- Data Modeling.
- Snowflake / Databricks.
- Git / GitHub.