DataOps Engineer
Description
We are looking for a highly motivated DataOps Engineer to join our Data team. In this role, you will be responsible for ensuring the stability, reliability, and quality of our data pipelines and internal data systems. You will work closely with Data Engineers, IT Infrastructure, IT Operations, and Data Analysts/Scientists to maintain data workflows, improve operational efficiency, and ensure data is delivered accurately and on time. This role is critical to sustaining data freshness, pipeline robustness, platform observability, and smooth operations across multiple products and internal backend tools.
Responsibilities
- Ensure the reliable and timely execution of daily data pipelines and scheduled workflows.
- Operate and maintain internal data services, including ingestion layers, OLAP/lake storage, materialised views, and task dependencies.
- Contribute to CI/CD workflows for data pipelines and participate in deployments, version management, and change control.
- Monitor orchestration systems (e.g., Airflow), troubleshoot pipeline failures, delays, and anomalies, and drive continuous performance improvements.
- Implement and maintain data quality checks, anomaly detection, schema validation, and audit processes.
- Collaborate with Data Engineers on table lifecycle management, storage optimisation, partitioning strategies, and schema evolution.
- Work with IT Infrastructure and IT Operations teams to improve platform observability, including logging, metrics, and alerting.
- Develop and maintain SOPs, platform standards, best practices, and troubleshooting documentation.
- Provide operational support to internal users (DE/DA/DS/Ops) for issues such as query performance, missing data, or inconsistent KPIs.
Requirements
- 2+ years of experience in Data Ops, Data Engineering, BI Engineering, or a similar operational data role.
- Experience with CI/CD workflows, Docker, Kubernetes, or other DevOps-related practices.
- Hands-on experience with workflow orchestration tools such as Airflow (or equivalent).
- Familiarity with mainstream data engineering technologies such as Kafka, Spark, Flink, Delta Lake, Iceberg, Hudi, ClickHouse, or Doris.
- Good understanding of data warehousing concepts, including partitioning, schema evolution, table lifecycle management, and OLAP vs. data lake architectures.
- Strong SQL skills and familiarity with Python for scripting, automation, or validation.
- Strong debugging and problem-solving skills, especially for data anomalies and pipeline failures.
- Comfortable working cross-functionally with DE/Infra/Ops/DA/DS teams in a fast-paced environment.
- Mandarin proficiency is preferred
