Role Overview
The Applied Machine Learning Ark team builds full‑stack LLM service platforms (MaaS) for US and global markets, developing agents, multimodal models, training, fine‑tuning, inference, alignment, and large‑scale log analytics pipelines.
Responsibilities
- Design evaluation systems for LLM‑based agents, measuring task success, tool use, reasoning quality, and reliability.
- Build benchmarks and automated judging pipelines using rule‑based checks, model‑based judging, and human review.
- Analyze agent execution traces and user feedback to uncover failure patterns and drive system improvements.
- Support the closed‑loop from experience to capability, collaborating with research, platform, and product teams to productionize methods.
Requirements
- Currently completing or recently completed a Bachelor’s or Master’s degree in Computer Science, AI, ML, Data Science, or related field.
- Solid foundation in machine learning and deep learning.
- Hands‑on experience with LLM‑based systems (agents, tool calling, retrieval, multi‑agent) via research, internships, or projects.
- Strong Python programming skills and experience with a mainstream ML or agent evaluation framework.
- Demonstrated research or engineering ability through publications, major projects, internships, or open‑source contributions.
- Preferred: Publications in top ML/NLP venues (NeurIPS, ICML, ICLR, ACL) on agent learning, self‑evolving systems, or evaluation.
- Preferred: Experience designing metrics, model‑based judging, annotation, or statistical analysis for evaluation.
- Preferred: Familiarity with LLM post‑training, reasoning, planning, or continual learning methods.
- Preferred: Experience with feedback‑driven optimization loops or large‑scale log/trace analysis.