Data Engineer

Responsibilities

• Develop and maintain ETL processes to extract, transform, and load data from various sources to target data repositories.
• Work closely with the Data Architect to understand data models and ensure that ETL processes align with the overall data architecture.
• Monitor and maintain data infrastructure to ensure data quality and accuracy, including implementing data security and privacy policies.
• Provide technical support to end-users to help them access and utilize data effectively.
• Work collaboratively with other members of the data team to ensure the effective and efficient use of data within the organization.
• Develop and maintain documentation of ETL processes and data infrastructure.

Qualifications

• Strong SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working with a variety of databases.
• Experience with dbt (data build tool) for data transformation, testing, and documentation.
• Experience building and optimizing 'big data' data pipelines, architectures and data sets on Databricks platform.
• Build processes supporting data transformation, data structures, metadata, dependency and workload management.
• A successful history of manipulating, processing and extracting value from large disconnected datasets.
• Experience supporting and working with cross-functional teams in a dynamic environment.
• Experience with object-oriented/object function scripting languages: Python, Java, Scala, C#, etc.
• Experience with Apache Airflow for workflow orchestration and pipeline monitoring.
• Experience in using Databricks, PySpark for big data processing.
• Previous work designing/managing data structures within a data warehouse.
• Experience in data stream processing and its processor is a plus.
• Experience developing application eg. web application, mobile application, REST web services is a plus.