Sarvam logo for SDE2 - API Platform job on DevsUnite

SDE2 - API Platform

Sarvam

Actively HiringFull-time
Apply
Location

Bengaluru, Karnataka, India

Experience

Not disclosed

Role

SDE2 - API Platform

Salary

Competitive

Share this job

Know someone who'd be a great fit?

Job Description

About The Role

Sarvam's model APIs (LLM, ASR, TTS, Vision, Translate) are how developers and enterprises ship products on our foundation models. Our OpenAI-compatible LLM APIs route every request across multiple model deployments, with streaming, failover, metering, billing, and rate-limiting all part of the same stack. Today it's FastAPI and Python; we're in the middle of rewriting it in Go.

We're hiring a Software Engineer (SDE 2) to own core parts of this platform. The features. The reliability. The performance. This is an IC role.

**What You'll Do

  • Building and evolving the LLM API platform, from the moment a request hits the edge to the response that goes back out
  • The Python→Go rewrite: building services in Go alongside the existing Python stack, without compromising reliability
  • Streaming infrastructure: SSE for Chat Completions and Responses, long-lived connections, clean handling of cancellations and client disconnects
  • Routing and failover: weighted provider chains, retries with backoff, and latency-driven admission control across model deployments
  • Protocol compatibility: keeping the API faithful to the OpenAI spec across inference servers (vLLM, SGLang, Dynamo) that each behave slightly differently
  • The commercial layer: metering, billing, and rate-limiting, engineered to the same bar as the model APIs themselves
  • Performance and reliability for a multi-tenant platform serving long-context agentic and coding workloads
  • Observability (logging, metrics, tracing), so the team can debug production confidently and quickly
  • Integration and load testing: the harness that lets us ship fast without breaking customer-facing APIs
  • Close partnership with the Inference and MLOps teams on production readiness for new model launches
  • Design-first, documentation-first engineering culture within the team: RFCs before code, decisions written down, no tribal knowledge

What We're Looking For

…

Required Skills

LLMsPythonFastAPIPostgreSQLRedis

Ready to apply for SDE2 - API Platform?

See how you stack up at Sarvam before you apply, or go straight to the application.

Apply Anyway