Role Overview
Postman is seeking an AI Engineer Intern to work with its AI team on large-scale AI and agentic systems, from data pipelines to production deployment. You will own discrete pieces of real systems under senior mentorship, not shadow work or isolated coursework-style projects.
Responsibilities
- Contribute to APIFlow-Bench: design and review benchmark tasks and mock API environments, extend the Python evaluation harness and task-generation pipeline, and help maintain the public multi-model leaderboard with statistical confidence intervals.
- Help build an action-level AI safety benchmark that scores what agents do in simulated enterprise API environments; work on scenario design, threat modeling (prompt injection, data exfiltration, permission overreach), and auditable evaluation.
- Fine-tune open-weight models for tool calling and agentic tasks using PyTorch (SFT, distillation, RL) on managed platforms and self-managed cloud GPUs; run rigorous experiments, ablations, and error analysis, and report results and cost honestly.
- Evaluate ultra-low-bit quantized models for on-device use; extend quantized vs. full-precision comparisons and analyze where and why they diverge.
- Help build Postman’s in-product AI agent (Agent Mode) with a minimal architecture that calls LLM APIs directly (tool loops, multi-step execution, checkpointing), primarily in TypeScript; read open-source agent harnesses and turn learnings into design specs and prototypes.
- Document experiments, design decisions, and runbooks; flag safety, fairness, or privacy concerns in model or agent behavior.
Requirements
- Currently pursuing a BS, MS, or PhD in Computer Science, Data Science, or a related quantitative field.
- Hands-on experience training or evaluating ML models through course projects, research, hackathons, or a prior internship.
- Solid Python fundamentals: data structures, functions, basic testing; comfortable writing and reviewing code outside notebooks.
- Working knowledge of at least one deep-learning framework (PyTorch preferred).
- Clear written and verbal communication, with a habit of documenting what you build.
Nice to Have
- Experience fine-tuning open-weight LLMs (SFT, LoRA, RL, or distillation) with benchmark-measured improvement.
- Experience building LLM agents (tool calling, multi-step loops) or LLM evaluation harnesses/benchmarks with statistical rigor.
- Track record shipping real software end-to-end: APIs and services, CLIs, Docker, CI/CD, cloud; public code on GitHub is a plus.
- Interest or experience in AI safety and robustness: red-teaming, prompt injection, agent security, fairness, or interpretability.
- Exposure to model-efficiency work: quantization, low-bit inference, or serving optimization.