Senior Data Engineer

About this role

ABOUT US

Dragonfly is building the world's first Automated Solutions Architect. We help businesses navigate the complex landscape of modern tools (SaaS, AI, Infrastructure) by using an AI-powered platform that understands their unique context and recommends the optimal tech stack.

Our platform is powered by a proprietary knowledge graph of 230K+ products, 4M+ companies, and the relationships between them — built from 100+ external data sources, LLM-driven research, and human curation. The architecture is designed but much of it still runs as manually-triggered pipelines. We need someone to turn it into a production data platform.

THE ROLE

We don't have a separate data team with its own stack. The data platform lives in the same monorepo as the product, follows the same engineering practices, and ships through the same CI pipeline. You'll be working alongside product engineers in Python and SQL — the same codebase, the same review process, the same standards.

The best way to describe this role is a software engineer who builds data systems. You'll be as comfortable making a product change as you are writing a pipeline.

We've designed the data architecture: a medallion-layer pipeline in BigQuery via Dataform, a signal-based quality scoring model, an append-only ontology, and an AI-driven research pipeline. We need someone to own it — productionise what exists, extend it as the product evolves, and keep it running day-to-day.

We operate a high-autonomy, high-trust environment. You'll be given a problem and the space to solve it — not a task list. AI is your first port of call for everything: understanding the codebase, exploring data, drafting implementations, debugging. Tickets describe what needs doing, not how. We expect you to think beyond the immediate task — consider knock-on effects, integration points, and how your work fits into the broader product and business strategy. Curiosity matters: you should want to understand how everything connects, not just the bit you're working on.

THE DATA PIPELINE

Dragonfly's core asset is a proprietary knowledge graph — every SaaS product, AI tool, and infrastructure service, what it does, who makes it, how it compares, and how confident we are in that data.

Building this knowledge graph is a multi-stage pipeline: raw data from external sources is normalised, scored for quality, matched and enriched by AI agents, curated by humans, and served to downstream systems (search, recommendations, product catalogue). The architecture follows a medallion pattern (Bronze → Silver → Gold) in BigQuery via Dataform, with an append-only ontology at the centre.

What's missing is the production engineering: scheduling, monitoring, incremental processing, source onboarding automation, and the platform layer that makes it all self-service for product engineers.

WHAT YOU'LL DO

1. PRODUCTION DATA PIPELINES

The pipeline exists as ~200 Dataform files across multiple datasets. Today, builds are triggered manually. Incremental processing has been validated in design but not hardened for production. You'll own making this run reliably.

- Scheduled builds, retry logic, and failure alerting

- Harden incremental processing — source refreshes need to flow through without full rebuilds

- Monitoring: volume checks per layer, freshness SLAs, broken source contract detection

- Maintain and evolve the data marts that feed search, recommendations, and the product catalogue

- Manage versioned dataset transitions with rollback capability

2. SOURCE ONBOARDING & INGESTION

We ingest from several sources. Adding a new source today means writing a silver view by hand. You'll build a repeatable process so that onboarding a new vendor is a config change, not a project.

- Evaluate and implement managed ingestion tooling (Airbyte, Meltano, or similar) for batch sources

- New sources land in Bronze and flow through the existing Silver standardisation layer automatically

- Handle the reality of messy source data — self-referential URLs, missing fields, schema drift, API rate limits

- Support three ingestion patterns: scheduled batch, ad-hoc triggers, and streaming from internal AI agents

3. DATA GOVERNANCE & LINEAGE

We've started with Dataplex for source trust metadata and quality scans. You'll extend this into a proper governance layer that's automated, not bureaucratic.

- Data contracts between pipeline layers (schema enforcement, freshness SLAs)

- Dataplex tagging for lineage tracking across the full pipeline

- Quality monitoring — volume checks, anomaly detection, broken source contract alerts

4. PLATFORM FOR PRODUCT ENGINEERS

You're an enabler, not a gatekeeper. Product engineers building the webapp, AI agents, and recommender system should be able to iterate on data without needing to understand the plumbing.

- Well-documented, stable data marts with clear contracts

- Self-service tooling so engineers can query, debug, and trace data through the pipeline

- Make it easy to add new derived fields, views, and downstream consumers

5. FEATURE ENGINEERING & DATA SCIENCE

The line between data engineering and data science is blurry here. Our scoring model, quality tiers, and recommendation features are the pipeline. You'll work closely with data science (and do some yourself).

- Own the quality scoring pipeline — signal computation, score aggregation, tier thresholds

- Iterate on signal weights and quality thresholds with empirical validation

- Build and maintain entity features that feed the recommender

- Exploratory analysis in BigQuery to inform pipeline design — the kind of deep-dive EDA that shapes architecture decisions

6. AI-NATIVE WORKFLOW

Our quality scoring model — designed, validated against real data, and documented extensively — was built using Claude Code to interrogate BigQuery directly. Work that would have taken a team weeks of testing took hours. This is how we build data systems: AI does the heavy…