Technical Director, Data Engineering

About this role

<p><span data-olk-copy-source="MessageBody">Diligent is looking for a Technical Director of Data Engineering to help design and build the next generation of data and AI infrastructure powering our products and internal platforms.</span></p> <p>This is not a traditional management role. We are looking for an experienced builder; someone who still enjoys writing code, debugging distributed systems, evaluating frameworks, and getting hands-on with architecture and implementation.</p> <p>You will work across large-scale ingestion pipelines, search systems, AI/LLM infrastructure, event-driven architectures, vector databases, analytics platforms, and real-time data processing. You should be equally comfortable discussing high-level architecture with senior leadership and diving into a failing Kubernetes pod or optimizing a Spark job.</p> <p>The ideal candidate has strong opinions informed by real-world experience, understands tradeoffs deeply, and can move quickly without creating unnecessary complexity.</p> <h3>What You’ll Do</h3> <ul> <li> <p>Design and build scalable data platforms and distributed processing systems</p> </li> <li> <p>Develop modern ingestion, transformation, and retrieval pipelines for structured and unstructured data</p> </li> <li> <p>Build systems supporting AI/LLM applications, semantic search, RAG pipelines, vector search, and agentic workflows</p> </li> <li> <p>Work hands-on with engineering teams to implement production-grade solutions rather than producing slideware</p> </li> <li> <p>Evaluate and standardize frameworks, tooling, and infrastructure patterns across teams</p> </li> <li> <p>Improve performance, reliability, observability, and cost efficiency of data systems</p> </li> <li> <p>Partner with product and platform engineering teams to accelerate delivery of AI-native capabilities</p> </li> <li> <p>Drive pragmatic engineering decisions balancing speed, maintainability, and operational simplicity</p> </li> <li> <p>Mentor engineers technically through design reviews, architecture guidance, and pair debugging</p> </li> <li> <p>Help establish engineering standards around CI/CD, testing, data quality, monitoring, and operational excellence</p> </li> </ul> <h3>What We’re Looking For</h3> <h3>Strong Hands-On Engineering Experience</h3> <p>Candidates should have significant real-world experience building and operating production systems using many of the following:</p> <h4>Data & Distributed Systems</h4> <ul> <li> <p>Airflow</p> </li> <li> <p>Elasticsearch / OpenSearch</p> </li> <li> <p>Vector databases and semantic retrieval systems</p> </li> <li> <p>MongoDB, PostgreSQL, DynamoDB, or similar platforms</p> </li> </ul> <h4>Cloud & Infrastructure</h4> <ul> <li> <p>AWS</p> </li> <li> <p>AWS CDK</p> </li> <li> <p>Serverless architectures</p> </li> <li> <p>Distributed observability and monitoring stacks</p> </li> </ul> <h4>AI / Search / Modern Data Applications</h4> <ul> <li> <p>LLM integration patterns</p> </li> <li> <p>RAG architectures</p> </li> <li> <p>Embeddings and vector search</p> </li> <li> <p>MCP servers and AI orchestration frameworks</p> </li> <li> <p>LangChain, LlamaIndex, DSPy, or similar ecosystems</p> </li> <li> <p>AI evaluation, tracing, and observability tooling</p> </li> <li> <p>Search relevance and ranking systems</p> </li> </ul> <h4>Backend Engineering</h4> <ul> <li> <p>Python strongly preferred</p> </li> <li> <p>Experience with Java, Go, or TypeScript is a plus</p> </li> <li> <p>API design and distributed service architectures</p> </li> <li> <p>Event-driven and asynchronous systems</p> </li> </ul> <h2>The Right Candidate</h2> <ul> <li> <p>Still enjoys building and debugging systems directly</p> </li> <li> <p>Has strong technical depth, not just architectural vocabulary</p> </li> <li> <p>Comfortable operating in ambiguity and fast-moving environments</p> </li> <li> <p>Understands how to simplify systems instead of endlessly abstracting them</p> </li> <li> <p>Has experience modernizing legacy platforms and evolving architectures incrementally</p> </li> <li> <p>Can distinguish between engineering fundamentals and hype cycles</p> </li> <li> <p>Values shipping working systems over theoretical perfection</p> </li> </ul> <h3>What Success Looks Like</h3> <ul> <li> <p>Engineering teams can move faster because the underlying platforms are reliable and scalable</p> </li> <li> <p>AI and data systems become production-grade rather than experimental prototypes</p> </li> <li> <p>Infrastructure costs and operational complexity are reduced through better architecture</p> </li> <li> <p>Search, ingestion, and retrieval systems improve significantly in performance and relevance</p> </li> <li> <p>Teams adopt consistent, maintainable…