Internship: Data engineering: Schema evolution 

Your future job

Data engineering: schema evolution handling on incoming data flow.
There's a constant increase (in volumes and ingestion rates) on incoming dataflows from various sources towards our data analytics and reporting platforms. Over time, the fields and/or their definitions within those flows can change.

Next to the development of schema evolution detection and handling scripts, these setups need to be embedded within the data lake environment in order to be used in an operational environment.

The goal of the project is to identify schema evolution on incoming data flows, to 'productize' this mechanism and to incorporate it on the data lake intake side.

- Schema evolution detection and handling on different types of data flows

- Productization of the solution (setting up data pipelines, versioning, pipelines for deploying different versions,...)
- Implement the mechanism in the data lake setup


(From 4 weeks up to 6 months)

Your profile

• Student in Bachelor or Master in IT 

• IT software analysis, design and development practices 

• Minimal: Python & SQL development

• Preferably: Git/Gitlab CI, Docker, Kubernetes 

• Data engineering knowledge

 

Main technologies used:

• Metadata solutions - catalogs, ...
• Databricks / spark
• Python, SQL
• Continuous delivery (GIT, gitlab CI, Docker, Kubernetes,..)
• Datalake concepts knowledge

Competencies the student could develop

• Working with state-of-the-art enterprise applications frameworks used to develop and deploy applications to be used worldwide
• Analysis and development methodologies like Domain Driven Design and continuous integration/ deployment
• Data engineering tasks like ETL, Python

 

We offer

• a challenging job in a dynamic high-tech international environment
• the opportunity to take ownership of your professional passion in order to contribute to the success of the company
• an enjoyable, team-oriented and professional atmosphere in a flat-structured organization
• versatile development opportunities