Role Overview
Sarvam's API platform team seeks an SDE2 to build and evolve the LLM API platform, handling routing, streaming, reliability, and performance.
Responsibilities
- Build and evolve the LLM API platform from edge request to response.
- Lead the Python→Go rewrite while maintaining reliability.
- Implement streaming infrastructure (SSE, long-lived connections, cancellation handling).
- Design routing and failover with weighted provider chains, retries, latency-driven admission control.
- Ensure OpenAI spec compatibility across inference servers (vLLM, SGLang, Dynamo).
- Build metering, billing, and rate-limiting for the commercial layer.
- Deliver performance and reliability for multi-tenant, long-context agentic workloads.
- Implement observability (logging, metrics, tracing) for production debugging.
- Develop integration and load testing harness for fast, safe releases.
- Collaborate with Inference and MLOps teams on production readiness for new models.
- Uphold design-first, documentation-first culture (RFCs, written decisions).
Requirements
- 3+ years building production backend systems.
- Strong Python in production: asyncio, FastAPI or similar, experience under real load.
- Working knowledge of Go or strong willingness to learn; work in both languages during rewrite.
- Experience building low-latency, multi-tenant backend services.
- Hands-on experience with streaming systems: SSE or WebSockets, long-lived connections, backpressure.
- Strong PostgreSQL and Redis fundamentals: schema design, query performance, caching, rate limiting.
- Comfort running services on Kubernetes in production.
- Ownership mindset: test before shipping, debug production with metrics/logs/traces, write RCAs.