Senior Data Engineer
Design, build, and implement generalized large-scale, sophisticated data pipelines using Nifi for downstream analytics and data science for our Sport Performance products. Design and develop scalable Nifi ingestion pipelines within AWS cloud services to consume real-time and batch data from external sources. Ensure the seamless integration of AWS-based tools for data storage, processing, and analytics. Responsible for ETL development and warehousing using Python and Java. Create data pipeline triggers and filters within ETL (extract, transform, and load) process to ensure appropriate optimization of data flowing through system and resource usage. Implement monitoring and error handling for all new parts of the data pipeline to ensure observability and alerting is available. Establish rigorous unit testing across the data pipeline to ensure robustness of the system. Design and create data models for use throughout the ETL system. Utilize Kafka to efficiently and to effectively store data to move throughout the data pipeline and for downstream data science and analytics usage. Design data architecture and data models for both internal and external representations of data. Build the data transforms within the data pipeline to convert data from external to internal representations. Conduct data analytics and debugging of bad data by writing SQL queries. Build automated cleaning of data to remove bad or unusable data from downstream consumers with logging to understand the frequency and depth of the underlying issues. Collaborate with other engineering teams to adopt standard methodologies, drive scalability, and increase consistency across systems. Maintain awareness of company standards and technology guidance; use JIRA, an Agile project mgmt. tool, to ensure efficient data development; collaborate with peers to align projects with overall direction. Follow best practices across Data Engineering to ensure scalable, consistent data architecture and system. Utilize Java language to build data processor in Nifi framework. Utilize Docker to ensure consistent, repeatable, and isolated environment for software development and testing. Work in a self-driven, independent fashion to meet Sport driven deadlines.
- Design, build, and implement generalized large-scale, sophisticated data pipelines using Nifi for downstream analytics and data science.
- Design and develop scalable Nifi ingestion pipelines within AWS cloud services to consume real-time and batch data from external sources.
- Ensure the seamless integration of AWS-based tools for data storage, processing, and analytics.
- Perform ETL development and warehousing using Python and Java.
- Create data pipeline triggers and filters within the ETL process to ensure appropriate optimisation of data flow and resource usage.
- Implement monitoring and error handling for all new parts of the data pipeline to ensure observability and alerting.
- Establish rigorous unit testing across the data pipeline to ensure robustness of the system.
- Design and create data models for use throughout the ETL system.
- Utilise Kafka to efficiently store and move data throughout the pipeline for downstream analytics.
- Design data architecture and data models for both internal and external representations of data.
- Build data transforms within the data pipeline to convert data from external to internal representations.
- Conduct data analytics and debugging of bad data by writing SQL queries.
- Build automated cleaning of data to remove unusable data from downstream consumers with logging.
- Collaborate with other engineering teams to adopt standard methodologies, drive scalability, and increase consistency across systems.
- Maintain awareness of company standards and technology guidance using JIRA to ensure efficient data development.
- Utilise Java language to build data processors in the Nifi framework.
- Utilise Docker to ensure consistent, repeatable, and isolated environments for software development and testing.
- Master’s degree in Computer Science, Computer Engineering, or closely related field (required).
- 1 year experience as a data engineer or related occupation (required).
- 1 year of experience with: Python (required).
- 1 year of experience with: Java (required).
- 1 year of experience with: Kafka (required).
- 1 year of experience with: AWS (required).
- 1 year of experience with: Docker (required).
- Experience with ETL Development and Warehousing (required).
- Experience with analytic and debugging using SQL (required).
- Experience in an Agile development environment (required).
- Experience designing data architecture (required).
- A collaborative environment with colleagues from all over the world (Offices in Europe, Asia and US) including various social events and teambuilding.
- Flexibility to manage your workday and tasks with autonomy.
- A balance of structure and autonomy to tackle your daily tasks.
- Vibrant and inclusive community, including Women in Tech and Pride groups which welcome all participants.
- Global Employee Assistance Programme.
- Calm and Reulay app (leading well-being apps designed to support focus, quality rest, mindfulness, and long-term mental resilience).
- Online training videos.
- Flexible working hours.
- Comprehensive benefits package.
- Performance bonus program.
- Equity stock purchase.
- 401k contribution.
Sportradar is a global sports technology company that collects, analyses and distributes sports data to betting operators, sports leagues and media companies. Headquartered in St. Gallen, Switzerland, and founded in 2001, it provides betting services, managed trading, integrity monitoring and audiovisual streaming across hundreds of sports. The company holds official data partnerships with organisations including the NBA, NHL, FIFA and ICC. Sportradar is listed on the Nasdaq stock exchange and employs several thousand people worldwide.