Role Overview
Cohere's Data Infrastructure team builds the unified storage layer feeding petabyte-scale model training across thousands of GPUs. This role involves designing, operating, and scaling distributed storage systems on Kubernetes, collaborating with researchers to meet throughput, latency, and durability requirements.
Responsibilities
- Design, build, and operate the distributed storage system feeding model training and evaluation.
- Run storage systems on Kubernetes clusters at petabyte scale.
- Partner with researchers and training-infra teams to translate data access patterns into throughput, latency, and durability requirements.
- Solve networking, I/O, and consistency challenges moving large datasets and checkpoints across regions and backends.
Requirements
- Strong storage fundamentals: replication, consistency, caching, data lifecycle management.
- Strong coding ability in Python and Go (experience in either, willingness to learn the other).
- Experience running stateful systems on Kubernetes (Persistent Volumes, CSI drivers, StatefulSets).
- Hands-on experience with cloud object storage (Amazon S3) and POSIX-style filesystems.
Nice to Have
- Experience with parallel/HPC filesystems (Weka, VAST, Lustre).
- Familiarity with data-loading and checkpointing patterns for large-scale model training.
- Interest in large-model training workflows, trajectory data, and evaluation workloads.
Benefits
- Weekly lunch stipend ($75/£75 local equivalent).
- Full health, dental, mental health budget; RRSP/401K/pension matching.
- 100% parental leave top-up for 6 months.
- 6 weeks paid vacation; enrichment stipends (arts, fitness, education, workspace).
- Remote-friendly with office access, co-working benefit, $500 home office stipend.