Staff SRE, AI Infrastructure

STAFF SRE, AI INFRASTRUCTURE Location: North America Remote / San Francisco · Full-Time ABOUT ANDROMEDA Andromeda Cluster was founded by Nat Friedman and Daniel Gross to give early-stage startups access to the kind of scaled AI infrastructure once reserved only for hyperscalers. Today, Andromeda works with leading AI labs, data centers, and cloud providers to deliver compute when and where it's needed most. Our aim is to become a liquidity layer for global AI compute — routing workloads across providers, GPU generations, and geographies the way financial markets route capital. We're a small, senior team where one engineer's judgment shapes every customer's experience. You'll join early enough to define how we run infrastructure at scale, work directly with the world's most demanding AI customers, and build a career operating at the frontier of what compute can do. THE ROLE We're hiring a Staff SRE to own the reliability of Andromeda's infrastructure end to end — from a node being racked and joined to a cluster, through the schedulers and control planes that place jobs on it, up to the customer-facing surface where a training run either succeeds or doesn't. We're looking for someone with multiple years of hands-on experience operating GPU infrastructure at scale. You read NVIDIA release notes the day they drop. You have war stories about NCCL, fabric topology choices, and what it takes to keep a multi-thousand-GPU run healthy. You move comfortably from a kernel-level perf trace to a customer incident bridge in the same hour, and you write the postmortem yourself. WHAT YOU'LL OWN - Highest-Priority Incident Leadership: Carry the pager. When a top-customer training run degrades or a multi-cluster incident hits, you're the engineer who walks the stack from PyTorch → NCCL → driver → fabric → hardware until the answer is found. You lead the response, write the postmortem, and ship the systemic fix. - Production Operations of GPU Fleets: Own the day-to-day health of thousands of GPUs across providers and generations. Node lifecycle, burn-in, validation, draining, repair workflows, firmware rollouts, driver upgrades — the unglamorous work that decides whether the platform actually holds up. - Observability & Health Systems: Build and own the telemetry, GPU health checks, fabric monitoring, and automated remediation that let us catch a degraded NVLink or a flaky transceiver before a customer does. Tooling lives on your laptop; you ship it. - On-Call Practice: Define how on-call works at Andromeda — rotations, escalation, runbooks, incident command, blameless review. As the team grows, you set the bar. - Customer-Facing Technical Presence: Be the senior reliability voice in the room with sophisticated AI infra customers and providers. Run incident reviews with a customer's principal engineer. Scope demanding workloads. Sit in on architecture deep-dives and deal cycles where reliability credibility closes the room. - Partnership with Engineering: Work shoulder-to-shoulder with the product team. You design with SLOs, error budgets, and failure modes in mind; they ship features; together you close the loop on every systemic issue. Translate customer pain into actionable priorities for product teams. - Hardware & Buildout Influence: Partner with providers and DC teams on physical design — rack and pod layout, power and cooling envelopes, network topology, burn-in and validation — to keep failure modes out of production before they arrive. - Mentorship as a Daily Practice: Spend real time every day making other engineers better. Incident reviews, pairing on diagnosis, written guidance, hiring. WHAT WE'RE LOOKING FOR - Years in This Space, Not Months: Multiple years building and operating large-scale GPU infrastructure as your primary job. You came up through this industry. - Staff-Level SRE Track Record: A clear history of owning the reliability of load-bearing infrastructure. You've been the senior engineer a team relies on when production is on fire and the failure mode is in a layer no one's touched yet. - GPU Systems Obsession: Deep, hands-on with NVIDIA H100/H200/B200/GB200 (or equivalent) at scale. You understand memory hierarchies, ECC and SBE/DBE behavior, thermal envelopes, NVLink and NVSwitch topology, and hardware failure modes from direct production experience. You also have opinions about what's coming next and why. - High-Performance Networking, in Production: Real production experience with InfiniBand, RoCE, and NVLink fabrics for distributed training. You can diagnose a slow all-reduce, find a degraded link in a fat-tree, reason about congestion control, and design topology for the workloads it'll actually carry. - Distributed Training Internals: Working knowledge of how large training jobs actually run — NCCL, CUDA, PyTorch distributed, FSDP, DeepSpeed, Megatron, and modern checkpointing/recovery patterns. When a 1,000+ GPU job stalls, you kno