Deskripsi Pekerjaan
Nexus Distributed Labs builds foundation-tier compute platforms for large-scale generative models. We are seeking an experienced Senior AI Infrastructure Engineer to architect, optimize, and scale our core GPU orchestration clusters and high-throughput data pipelines.
In this role, you will interface directly with systems architects and ML researchers to minimize training bottlenecks, implement automated failover protocols, and maximize hardware utilization across thousands of accelerators. We offer tier-one compensation, equity participation, comprehensive healthcare, and flexible hybrid arrangements.
Tanggung Jawab
- Architect and maintain bare-metal and cloud Kubernetes clusters managing distributed H100/A100 GPU fabrics.
- Build fault-tolerant distributed storage and data ingestion pipelines sustaining multi-terabit throughput.
- Optimize network topologies utilizing InfiniBand and RoCE protocols to eliminate distributed training latency.
- Instrument deep-telemetry monitoring across hardware, orchestrators, and model runtimes via Prometheus and Grafana.
- Implement automated cluster recovery, preemptible workload rescheduling, and checkpointing routines.
- Partner with ML teams to profile model kernels, isolate runtime overhead, and increase MFU.
Kualifikasi
- Minimum 5 years of experience deploying and operating production distributed systems at scale.
- Deep production experience with Kubernetes, Linux kernel internals, and distributed file systems (e.g., Lustre, GPFS).
- Strong proficiency in Go, Python, or Rust for low-level systems automation and tooling.
- Direct operational knowledge of high-performance networking protocols (InfiniBand, RDMA, SR-IOV).
- Hands-on experience diagnosing hardware, thermal, and driver issues across modern accelerator fleets (NVIDIA CUDA stack).
- Proven record of owning infrastructure reliability under high-load mission-critical conditions.