Senior Data Engineer

About this role

<div class="content-intro"><p class="p1"><span style="font-size: 12pt;"><strong>About Us</strong></span></p> <p class="p1"><span style="font-size: 12pt;">Abacus Insights is transforming how data works for health plans. Our mission is simple: make healthcare data usable, so the people responsible for care and cost decisions can act faster, with confidence.</span></p> <p class="p1"><span style="font-size: 12pt;">We help health plans break down data silos to create a single, trusted data foundation. That foundation powers better decisions—so plans can improve outcomes, reduce waste, and deliver better experiences for members and providers alike. Backed by $100M from top investors, we’re tackling big challenges in an industry that’s ready for change. Our platform enables GenAI use cases by delivering clean, connected, and reliable healthcare data to support automation, prioritization, and decision workflows—and it’s why we are leading the way.</span></p> <p class="p1"><span style="font-size: 12pt;">Our innovation begins with people. We are bold, curious, and collaborative—because the best ideas come from working together. We embrace the thoughtful use of AI and automation to drive innovation and efficiency, and we look for individuals who are curious and adaptable—those excited to leverage emerging technologies to enhance how we work—while keeping human insight, connection, and our clients at the center of every decision.</span></p> <p class="p1"><span style="font-size: 12pt;">Ready to make an impact? Join us and let’s build the future together.</span></p></div><p><span style="font-size: 12pt; font-family: helvetica, arial, sans-serif;"><strong>About the role</strong></span></p> <p><span style="font-size: 12pt; font-family: helvetica, arial, sans-serif;">We are seeking an accomplished Data Engineer to join our dynamic and rapidly expanding Tech Ops division. With significant projected growth, this is an opportunity to drive meaningful technical impact. In this role, you will work directly with customers, data vendors, and internal engineering teams to design, implement, and optimize complex data integration solutions within a modern, large‑scale cloud environment.</span></p> <p><span style="font-size: 12pt; font-family: helvetica, arial, sans-serif;">You will leverage advanced skills in distributed computing, data architecture, and cloud-native engineering to enable scalable, resilient, and high‑performance data ingestion and transformation pipelines. As a trusted technical advisor, you will guide customers in adopting Abacus’s core data management platform and ensure high-quality, compliant data operations across the lifecycle.</span></p> <p><span style="font-size: 12pt; font-family: helvetica, arial, sans-serif;"><strong>Your day to day</strong></span></p> <ul> <li style="font-size: 12pt; font-family: helvetica, arial, sans-serif;"><span style="font-size: 12pt; font-family: helvetica, arial, sans-serif;">Architect, design, and implement high-volume batch and real-time data pipelines using PySpark, SparkSQL, Databricks Workflows, and distributed processing frameworks.</span></li> <li style="font-size: 12pt; font-family: helvetica, arial, sans-serif;"><span style="font-size: 12pt; font-family: helvetica, arial, sans-serif;">Build end‑to‑end ingestion frameworks integrating with Databricks, Snowflake, AWS services (S3, SQS, Lambda), and vendor data APIs, ensuring data quality, lineage, and schema evolution.</span></li> <li style="font-size: 12pt; font-family: helvetica, arial, sans-serif;"><span style="font-size: 12pt; font-family: helvetica, arial, sans-serif;">Develop data modeling frameworks, including star/snowflake schemas and optimization techniques for analytical workloads on cloud data warehouses.</span></li> <li style="font-size: 12pt; font-family: helvetica, arial, sans-serif;"><span style="font-size: 12pt; font-family: helvetica, arial, sans-serif;">Lead technical solution design for health plan clients, creating highly available, fault-tolerant architectures across multi-account AWS environments.</span></li> <li style="font-size: 12pt; font-family: helvetica, arial, sans-serif;"><span style="font-size: 12pt; font-family: helvetica, arial, sans-serif;">Translate complex business requirements into detailed technical specifications, engineering artifacts, and reusable components.</span></li> <li style="font-size: 12pt; font-family: helvetica, arial, sans-serif;"><span style="font-size: 12pt; font-family: helvetica, arial, sans-serif;">Implement security automation, including RBAC, encryption at rest/in transit, PHI handling, tokenization, auditing, and compliance with HIPAA and SOC 2 frameworks.</span></li> <li style="font-size: 12pt; font-family: helvetica, arial, sans-serif;"><span style="font-size: 12pt; font-family: helvetica, arial, sans-serif;">Establish and enforce data engineering best practices, such as CI/CD for data pipelines, code versioning, automated testing, orchestration, logging, and observability patterns.</span></li> <li style="font-size: 12pt; font-family: helvetica, arial, sans-serif;"><span style="font-size: 12pt; font-family: helvetica, arial, sans-serif;">Conduct performance profiling and optimize compute costs,…