Role Overview
Cohere seeks a software engineer to build and operate an ML workload platform that schedules training, evaluation, and reinforcement‑learning jobs on Kubernetes. The role includes a compensated 24x7 on‑call rotation and requires close collaboration with AI researchers and ML engineers. You will design job lifecycles, networking, runtime isolation, and storage for checkpoints while ensuring system reliability. The platform will serve enterprise customers building AI systems.
Responsibilities
- Own the platform that schedules and runs training, evaluation, and reinforcement‑learning jobs on Kubernetes.
- Build networking and runtime isolation, including agent and sandbox environments.
- Collaborate with data infrastructure to manage storage for checkpoints and training artifacts.
- Troubleshoot production failures and performance issues to improve system reliability.
- Create automation, infrastructure‑as‑code, and observability to keep the platform maintainable.
Requirements
- Experience building job platforms, batch systems, or Kubernetes controllers in production.
- Proven work on ML training workflows, RL sandboxes, container networking (CNI), or checkpoint storage at scale.
- Strong Python or Go programming skills; ability to extend open‑source projects.
- Distributed systems judgment, including retries and partial failure handling.
- Collaboration with researchers and ML engineers with aligned but distinct goals.
Skills
- Machine Learning
- Kubernetes
- Python
- Reinforcement-learning
- Go
Location
Remote position based in Canada.