Technical Director, Data Engineering

About this role

Diligent is looking for a Technical Director of Data Engineering to help design and build the next generation of data and AI infrastructure powering our products and internal platforms.

This is not a traditional management role. We are looking for an experienced builder; someone who still enjoys writing code, debugging distributed systems, evaluating frameworks, and getting hands-on with architecture and implementation.

You will work across large-scale ingestion pipelines, search systems, AI/LLM infrastructure, event-driven architectures, vector databases, analytics platforms, and real-time data processing. You should be equally comfortable discussing high-level architecture with senior leadership and diving into a failing Kubernetes pod or optimizing a Spark job.

The ideal candidate has strong opinions informed by real-world experience, understands tradeoffs deeply, and can move quickly without creating unnecessary complexity.

What You’ll Do

•
Design and build scalable data platforms and distributed processing systems

•
Develop modern ingestion, transformation, and retrieval pipelines for structured and unstructured data

•
Build systems supporting AI/LLM applications, semantic search, RAG pipelines, vector search, and agentic workflows

•
Work hands-on with engineering teams to implement production-grade solutions rather than producing slideware

•
Evaluate and standardize frameworks, tooling, and infrastructure patterns across teams

•
Improve performance, reliability, observability, and cost efficiency of data systems

•
Partner with product and platform engineering teams to accelerate delivery of AI-native capabilities

•
Drive pragmatic engineering decisions balancing speed, maintainability, and operational simplicity

•
Mentor engineers technically through design reviews, architecture guidance, and pair debugging

•
Help establish engineering standards around CI/CD, testing, data quality, monitoring, and operational excellence

What We’re Looking For

Strong Hands-On Engineering Experience

Candidates should have significant real-world experience building and operating production systems using many of the following:

Data & Distributed Systems

•
Airflow

•
Elasticsearch / OpenSearch

•
Vector databases and semantic retrieval systems

•
MongoDB, PostgreSQL, DynamoDB, or similar platforms

Cloud & Infrastructure

•
AWS

•
AWS CDK

•
Serverless architectures

•
Distributed observability and monitoring stacks

AI / Search / Modern Data Applications

•
LLM integration patterns

•
RAG architectures

•
Embeddings and vector search

•
MCP servers and AI orchestration frameworks

•
LangChain, LlamaIndex, DSPy, or similar ecosystems

•
AI evaluation, tracing, and observability tooling

•
Search relevance and ranking systems

Backend Engineering

•
Python strongly preferred

•
Experience with Java, Go, or TypeScript is a plus

•
API design and distributed service architectures

•
Event-driven and asynchronous systems

The Right Candidate

•
Still enjoys building and debugging systems directly

•
Has strong technical depth, not just architectural vocabulary

•
Comfortable operating in ambiguity and fast-moving environments

•
Understands how to simplify systems instead of endlessly abstracting them

•
Has experience modernizing legacy platforms and evolving architectures incrementally

•
Can distinguish between engineering fundamentals and hype cycles

•
Values shipping working systems over theoretical perfection

What Success Looks Like

•
Engineering teams can move faster because the underlying platforms are reliable and scalable

•
AI and data systems become production-grade rather than experimental prototypes

•
Infrastructure costs and operational complexity are reduced through better architecture

•
Search, ingestion, and retrieval systems improve significantly in performance and relevance

•
Teams adopt consistent, maintainable…