Senior Data Platform Engineer

Over deze functie

**About the Role** You support the design of client's multi-user data and analysis platform, in particular the storage layer component (including the data lake, which forms the basis of the data platform). You also support interactions with other components ('ingestion layer', 'core systems', 'front-end-systems'). You specialize in data and analytics platforms, with expertise in lakehouse architectures, Kubernetes-based data science environments, open source ecosystems, and European digital sovereignty initiatives. You have experience in designing scalable, interoperable, secure (GDPR, NIS2) and production-oriented platforms using technologies such as PostgreSQL, MinIO, OpenStack, Apache, Kubernetes, Docker, Spark, Onyxia, and modern governance and metadata frameworks. You will be the link between the architecture at the organizational level and the detailed technical implementation. You collaborate with internal departments, technical teams and external partners to find an integrated and robust solution. **Responsibilities** \- Propose, evaluate, test and provide advice on technologies, tools and methodologies for the 'storage' component (lakehouse). This includes: · BLOB storage, data lake · data warehouse relational database management system (RDBMS) · SQL RDBMS · NoSQL Database System. \- Propose how to implement the 'Medallion' architecture (bronze, silver, gold), archiving, and back-up within the data platform, taking into account the ingestion, processing, integration, consumption and dissemination of the different types of data flows and the obligations (legal, Belgian health data ecosystem, animals and environment). \- Evaluate, propose and test that these solutions are consistent with the general architecture of the platform, are based on open source, sovereign and multi-user technologies, and are compatible and interoperable with the other 'Application architecture' components, in particular with: · 'ingestion layer' (including the different sources) · 'core systems' · 'front-end-systems' (data science workbench, BI, data service gateway, secure processing environment) \- Ensure that the architecture and data flows comply with the requirements defined by the Data Office, GDPR, NIS2, Information Security Committee (ISC), the new registry policy and the European Health Data Space (health data holder, trusted data holder). \- Proactively solve challenges related to system choices and implementation \- Advise technical teams, evaluate technical designs and code, support best practices; \- Collaborate with the various departments involved, management and external partners and act as a technical contact person; \- Write (technical) documentation \- Ensure high-quality technical implementation

**Technical Requirements** · At least 10 years of experience in data platforms, data engineering, solution architecture, data storage, big data ecosystems, data modeling, and database technologies; · Design and implementation of modern data and analytics platforms in lakehouse architectures, open source ecosystems (without single vendor lock in), and European digital sovereignty initiatives. · Data Platform Architecture : data ingestion, data lakes, data warehousing, ETL/ELT pipeline · ...