Deskripsi Pekerjaan
Synthetix Labs is seeking an exceptional Senior AI Infrastructure Engineer to architect, scale, and optimize the distributed computing clusters powering our foundation models. In this role, you will eliminate compute bottlenecks, manage thousands of accelerator nodes, and deliver high-reliability distributed training pipelines.
You will collaborate closely with research scientists and platform engineers to design resilient data ingestion frameworks, fault-tolerant checkpointing systems, and automated cluster self-healing mechanisms.
Tanggung Jawab
- Architect, deploy, and manage multi-node accelerator clusters running Kubernetes and Slurm across hybrid-cloud environments.
- Optimize distributed training frameworks including PyTorch FSDP, DeepSpeed, and Megatron-LM for high GPU utilization and minimal network overhead.
- Design resilient data ingestion, caching, and checkpointing pipelines capable of handling petabyte-scale datasets.
- Automate infrastructure provisioning, cluster maintenance, and bare-metal deployment using Terraform, Ansible, and GitOps workflows.
- Build end-to-end telemetry and alerting for hardware health, InfiniBand fabrics, and thermal performance using Prometheus and Grafana.
- Diagnose low-level systems bottlenecks spanning kernel parameters, PCIe topology, and network fabrics.
Kualifikasi
- 5+ years of experience operating production Linux environments and high-performance compute clusters.
- Deep production experience orchestrating large-scale GPU workloads with Kubernetes.
- Practical knowledge of high-speed interconnects including InfiniBand, RoCE, and high-performance filesystems like Lustre or GPFS.
- Proficiency in Python and Go for systems automation, CLI tooling, and observability integration.
- Familiarity with CUDA, NCCL, and hardware telemetry suites such as NVIDIA DCGM.
- Bachelor's degree in Computer Science, Computer Engineering, or equivalent practical experience.