Lead Data Engineer

About the Role

We are looking for a visionary and technical Lead Data Engineer to guide and scale our data engineering team within our regional data chapter. You will own the architecture, evolution, and reliability of our next-generation regional data platform, optimize a Databricks-driven Data Lakehouse architecture, drive DataOps practices, implement robust data governance, and collaborate across functional squads to enable advanced analytics, business intelligence, and AI initiatives.

Responsibilities

• Design, build, and continuously optimize a scalable Data Lakehouse platform leveraging Databricks and AWS to support global business expansion.
• Lead design and implementation of automated real-time and batch ETL/ELT frameworks and pipelines; oversee integration with internal microservices, external insurance partners, and third-party APIs.
• Champion DataOps by building framework controls, schema registries, automated testing, and CI/CD pipelines for data assets using tools like dbt and Airflow; drive Databricks serverless migrations and automated performance monitoring.
• Own regional data quality, data observability (e.g., Elementary), data freshness, and data catalogs; ensure data security, PDPA compliance, and sensitivity tagging across multi-region boundaries.
• Collaborate with Executives, Product Owners, Software Developers, Data Analysts, and MLOps/Data Science squads to unblock technical dependencies and deliver actionable data products.
• Research and lead proofs-of-concept for emerging technologies such as Generative AI/Agentic AI data pipelines (automated knowledge bases, smart web scraping) into the data ecosystem.
• Manage, mentor, and elevate technical capabilities of junior and senior data engineers; ensure standardized practices and strong technical ownership across regional squads.

Qualifications

• 5+ years of experience in Data Engineering, Data Architecture, or related technical roles, with at least 2+ years leading engineering teams or core technical projects.
• Deep hands-on experience designing and managing production workloads in Databricks (Delta Lake, Unity Catalog, serverless paradigms).
• Master-level proficiency in SQL (complex queries, optimization, macros) and programmatic data engineering in Python or Scala.
• Extensive experience with Apache Spark and Big Data open-source frameworks.
• Experience with cloud data pipeline orchestration tools (e.g., Airflow, Dagster) and transformation tools like dbt.
• Solid AWS cloud expertise (S3, EC2, RDS, VPC, networking) integrated within data ecosystems.
• Expertise in data modeling and architecture, transactional databases, distributed storage, message queuing/streaming (e.g., Kafka), and Lakehouse patterns (Medallion architecture).
• Proven problem solving and systems thinking: root cause analysis on production infrastructure failures, complex migrations, and optimizing pipeline issues (e.g., small file storage).
• Bachelor’s or Master’s degree in Computer Science, Computer Engineering, Information Technology, or a quantitative field.

Skills

• Design and operation of Databricks-backed Data Lakehouse (Delta Lake, Unity Catalog, serverless).
• ETL/ELT development for real-time and batch data pipelines.
• Data governance, observability, security, and compliance (PDPA) across multi-region deployments.