Senior Data Engineer

About this role

<h1><span style="font-size: 12pt;">About Aiviq</span></h1> <p>Aiviq is a cutting-edge fintech company revolutionising financial services and asset management. We empower the world's leading asset managers with data-driven insights and innovative technology solutions. Our cloud-based platform transforms complex financial data into actionable intelligence, addressing critical challenges in client data quality and insights. Serving global asset managers overseeing trillions in Assets under Management, we're at the forefront of financial technology innovation.</p> <p><strong>Reporting Structure</strong></p> <p><strong>Reports to:</strong> Data Centre of Excellence Team Lead<br><strong>Dotted line to:</strong> Head of Engineering<br><strong>Location:</strong> UK (Hybrid)</p> <p><strong>Job Purpose</strong></p> <p>We're seeking an accomplished Data Engineer to join our Data Centre of Excellence while working closely with our Engineering team on our sophisticated financial data management platform. This role combines the technical depth of enterprise data engineering with the fast-paced delivery demands of product development, requiring someone who is adept at translating business logic into code, can think architecturally while attending to implementation details. You'll be the bridge between our data architecture standards and practical product delivery, ensuring our financial data pipelines are robust, performant, and built on solid engineering principles.</p> <p><strong>Key Responsibilities</strong></p> <p><strong>Data Engineering & Development</strong></p> <ul> <li>Design, build, and optimize data pipelines across Microsoft SQL Server and Azure Synapse Analytics environments</li> <li>Develop and maintain Spark SQL notebooks for complex data transformations and analysis</li> <li>Translate business logic and financial calculation requirements into clear, maintainable code</li> <li>Create data integrity checking scripts and validation frameworks in collaboration with QA teams</li> <li>Implement automated data quality checks and reconciliation processes</li> <li>Assist with the maintenance of a curated, anonymized dataset for system testing that covers all known scenarios and edge cases</li> <li>Analyse production datasets to identify anomalies, debug stored procedures and notebooks, and resolve data quality issues</li> <li>Demonstrate tenacity in investigating root causes, diving deep into complex problems until resolution is achieved</li> </ul> <p><strong>Architecture & Performance</strong></p> <ul> <li>Consult on database architecture decisions, balancing performance, scalability, and maintainability</li> <li>Optimize query performance and data processing workflows for large-scale financial datasets</li> <li>Design and implement solutions using Azure Data Factory, Delta Lake, and related technologies</li> <li>Think end-to-end about data flows while ensuring rigorous attention to implementation details</li> </ul> <p><strong>Documentation & Process</strong></p> <ul> <li>Create and maintain comprehensive documentation of database schemas, processes, and data flows</li> <li>Develop visual process models using tools such as Lucidchart, Visio, dbt, Azure Purview, or similar platforms</li> <li>Document data transformation logic and calculation methodologies for audit and compliance purposes</li> <li>Contribute to data governance standards and best practices across the organization</li> </ul> <p><strong>Production Support & Collaboration</strong></p> <ul> <li>Act as first point of escalation for high-priority data issues in production environments</li> <li>Partner with test automation engineers to develop data-driven testing strategies and create data integrity checking scripts</li> <li>Collaborate across engineering teams using Azure DevOps for CI/CD pipeline development</li> <li>Support both Data CoE initiatives and product engineering priorities through effective stakeholder management</li> </ul> <p><strong>Required Skills & Experience</strong></p> <p><strong>Technical Expertise</strong></p> <ul> <li><strong>Database Technologies:</strong> Strong proficiency in MS SQL Server and Azure Synapse Analytics</li> <li><strong>Big Data Processing:</strong> Hands-on experience with PySpark, Spark SQL, and notebook-based development</li> <li><strong>Cloud Platforms:</strong> Demonstrable experience with Azure ecosystem (Synapse, Data Factory, Delta Lake)</li> <li><strong>Programming:</strong> Solid coding skills in SQL, Python, and/or C#</li> <li><strong>Version Control:</strong> Experience with Git and Azure DevOps or similar CI/CD platforms</li> <li><strong>Testing:</strong> Experience building out unit and integration test frameworks and processes to ensure pipelines and notebooks and other code artefacts are fully automation-tested</li> </ul> <p><strong>Domain Knowledge</strong></p> <ul> <li>Ideally a proven track record working with complex financial data and calculations</li> <li>Understanding of financial data structures, reconciliation processes, and audit requirements</li> <li>Experience handling temporal data, slowly…