Lead Data Engineer

About the Company

Join a company that is pushing the boundaries of what is possible. We are renowned for our technical excellence and leading innovations, and for making a difference to our clients and society. Our workplace embraces diversity and inclusion – it's a place where you can grow, belong and thrive.

About the Role

We are seeking a Lead Data Engineer with 5+ years of big data and cloud data architecture expertise to design and modernize data platforms for large-scale transformation programs. This role combines deep technical hands-on expertise with architectural leadership, requiring mastery of big data frameworks (Spark, Flink, Kafka), cloud data platforms (BigQuery, Dataplex, Dataproc, Dataflow), and modern data stack technologies (Trino, Cloud Composer/Airflow, BigLake). The successful candidate will own data architecture design for both lift-and-shift migrations and cloud-native modernization, establish data governance frameworks, and provide technical leadership to engineering teams.

Key Responsibilities

• Design target data architectures for OSS lift-and-shift and cloud-native modernization scenarios. Make architectural decisions balancing performance, cost, scalability, and operational excellence.
• Own migration approach for big data platforms including HDFS to GCS migration, Hadoop ecosystem modernization, and data warehouse transformations. Establish phased migration patterns and validation strategies.
• Design BigQuery, Dataplex, BigLake, and data lakehouse architectures. Establish data organization patterns, access controls, and metadata management. Optimize for cost, query performance, and data discovery.
• Design Spark, Flink, and Dataflow pipelines for batch and streaming data processing. Establish processing patterns, optimization strategies, and cost management approaches.
• Design Cloud Composer and Airflow DAGs for complex data workflows. Establish orchestration patterns, error handling, monitoring, and retry strategies.
• Design data governance frameworks including data ownership, access controls, and metadata standards. Establish data lineage tracking, data catalogs, and governance policies.
• Establish data quality frameworks and validation strategies for migrations. Define quality rules, reconciliation criteria, and acceptance thresholds. Implement data quality monitoring and alerting.
• Design migration strategies for Hive Metastore to BigLake Metastore/Dataplex. Manage table schema migration, partition strategy optimization, and metadata preservation.
• Design query engine architectures supporting multiple data sources. Establish Trino/Presto configurations for federated query access.
• Design Kafka-based streaming architectures for real-time data ingestion. Establish Kafka topics, partition strategies, and consumer patterns.
• Design scalable data ingestion patterns for structured and unstructured data. Establish ELT/ETL frameworks optimized for cloud platforms.
• Design GCS-based storage architectures for data lakes. Establish data organization, partitioning, and lifecycle policies. Optimize storage costs and access patterns.
• Design modern lakehouse architectures combining data lake and warehouse capabilities. Establish table formats, schema management, and ACID transaction support.
• Provide technical leadership to data engineering teams. Mentor engineers on data architecture patterns, best practices, and technical decision-making. Conduct design reviews and architecture discussions.
• Establish code quality standards, design patterns, and testing frameworks. Review data pipeline code and designs.
• Optimize data pipelines for performance and cost. Profile and tune Spark, Dataflow, and BigQuery workloads. Establish cost monitoring and optimization practices.
• Create comprehensive data architecture documentation and design guides. Document migration approaches, implementation patterns, and operational procedures.

Required Qualifications

• Minimum 5 years of professional data engineering experience
• Minimum 5+ years of big data architecture and cloud data platform experience
• Expert-level proficiency in Apache Spark for large-scale data processing
• Strong experience with GCP data platforms (BigQuery, Dataproc, Dataflow, Dataplex, BigLake)
• Hands-on experience designing and implementing data pipelines and ETL/ELT processes
• Strong knowledge of data lake and lakehouse architectures
• Proficiency with workflow orchestration tools (Apache Airflow, Cloud Composer)
• Experience with big data technologies (Hadoop, HDFS, Hive, Kafka) and modernization approaches
• Understanding of data governance, data lineage, and metadata management
• Proficiency in Python and/or Scala for data engineering