Staff HPC Engineer

About this role

<div class="content-intro"><p>Biohub is the first large-scale initiative bringing frontier AI models, massive compute, and frontier experimental capabilities under one roof. We're building a general-purpose system to accelerate scientific discovery, integrating frontier AI models, biological foundation models, and lab capabilities, with the ultimate goal of curing disease. Our technology powers scientists around the world, translating AI capabilities into tools that accelerate research everywhere.</p></div><h2><span style="color: rgb(40, 40, 39); font-family: helvetica, arial, sans-serif;">The Team</span></h2> <p><span style="font-family: helvetica, arial, sans-serif;">The HPC Engineering team is part of the AI Compute Platform organization at Biohub, a non-profit research lab committed to open science and open-source AI. We own the design, operation, and reliability of hybrid GPU AI clusters that power frontier AI biology research: protein language models, genomic foundation models, and scientific reasoning systems built to be shared. Our  infrastructure  supports day-to-day AI researcher workflows. The team works at the intersection of AI tooling, distributed systems, HPC, and frontier AI, debugging deep AI infrastructure problems and building AI systems critical to the entire AI organization.</span></p> <h2><span style="color: rgb(40, 40, 39); font-family: helvetica, arial, sans-serif;">The Opportunity</span></h2> <p>We seek a Staff HPC Engineer to help lead the evolution of our advanced computing infrastructure into a next-generation hybrid HPC and AI platform. This role will help shape strategy, architecture, and operations for high-performance computing resources — including cutting-edge GPUs, large-scale storage, and high-speed networks — while enabling transformative science through AI and machine learning at scale.</p> <p>You will design, implement, and optimize a unified HPC-AI ecosystem blending on-prem Slurm-managed clusters, cloud GPU resources, and containerized environments. This hybrid environment will power everything from traditional HPC workloads to large AI training jobs, generative model development, real-time inference, and data-intensive pipelines.</p> <p>The successful candidate will be a thought leader in HPC infrastructure , capable of partnering with scientists, computational biologists, and software engineers to translate complex research needs into high-impact computing solutions. You will also foster adoption of emerging AI tools, and ensure our systems can scale to meet the demands of next-generation biomedical research.</p> <h2><span style="color: rgb(40, 40, 39); font-family: helvetica, arial, sans-serif;">What You'll Do</span></h2> <p>HPC  Engineering</p> <ul> <li>Build and support a hybrid HPC-AI environment with large-scale on-prem compute/storage and elastic cloud GPU clusters (Coreweave, AWS, GCP).</li> <li>Architect and optimize environments for large-scale AI training and tuning, and low-latency scientific workloads.</li> <li>Integrate MLOps and model deployment pipelines into HPC infrastructure, ensuring reproducibility and efficiency.</li> <li>Implement advanced resource scheduling and orchestration (Slurm, Kubernetes, SUNK) optimized for mixed HPC and AI workflows.</li> </ul> <p>Operational Excellence</p> <ul> <li>Support researchers with job optimization, GPU utilization best practices, and performance tuning for AI and HPC applications.</li> <li>Evaluate, deploy, and maintain AI/ML software stacks (e.g., PyTorch, TensorFlow, Hugging Face, RAPIDS) and HPC toolchains.</li> <li>Ensure robust data ingest, analysis, and management capabilities for AI and HPC workloads, including integration with parallel file systems and object storage.</li> </ul> <p>Collaboration & Enablement</p> <ul> <li>Work with diverse science teams to translate research requirements into hardware/software solutions, from experimental design through publication.</li> <li>Promote best practices for AI model training, validation, and deployment in shared computing environments.</li> <li>Foster a culture of shared learning by running internal workshops on HPC-AI tooling (e.g., VS Code remote dev, containerization, MLOps workflows).</li> </ul> <h2><span style="color: rgb(40, 40, 39); font-family: helvetica, arial, sans-serif;">What You'll Bring</span></h2> <p>Essential</p> <ul> <li>Bachelor’s or advanced degree in Computer Science, AI/ML, Data Science, Systems Engineering, or related field.</li> <li>10+ years building and managing HPC infrastructure, with significant experience integrating AI/ML workloads.</li> <li>Proven track record architecting environments for large-scale GPU AI training and inference in hybrid on-prem/cloud environments.</li> <li>Deep expertise with HPC scheduling (Slurm), container orchestration (Kubernetes), and cloud GPU services.</li> <li>Strong hands-on experience with AI frameworks (PyTorch, TensorFlow, JAX) and distributed training strategies (Horovod, DeepSpeed, Ray).</li> <li>Knowledge of MLOps best practices, including CI/CD for ML, model registry, experiment tracking, and performance monitoring.</li> <li>Exceptional ability to collaborate with multidisciplinary teams and communicate complex technical concepts clearly.</li> <li>Demonstrated leadership in guiding infrastructure teams, influencing organizational strategy, and fostering adoption of new…