Role Overview
The Software Engineer, ML Workload Platform will own the platform that schedules and runs training, evaluation, and reinforcement‑learning jobs on Kubernetes, ensuring capacity, failure handling, and observability for Cohere's foundation models.
Responsibilities
- Design and implement job scheduling, queueing, and multi‑tenant fairness on Kubernetes.
- Build networking and runtime isolation, including agent and sandbox environments.
- Collaborate with data infrastructure to provide storage for checkpoints and training artifacts.
- Troubleshoot production failures and performance issues, improving system reliability.
- Develop automation, infrastructure‑as‑code, and observability tooling.
Requirements
- Proven experience running a job platform, batch system, or Kubernetes controllers in production.
- Demonstrated distributed systems judgment: retries, partial failure, and automation trade‑offs.
- Experience with ML training workflows, RL sandboxes, container networking (CNI), or checkpoint storage at scale.
- Proficiency in Python or Go; preference for extending open‑source projects.
- Ability to collaborate with researchers and ML engineers with differing goals.