Role Overview
The Applied Machine Learning Ark team builds LLM service platforms offering Model-as-a-Service (MaaS) solutions, spanning LLM algorithms, training/fine-tuning/inference, prompt engineering, alignment, and intelligent agent systems. The US team operates large-scale log analytics pipelines and data-driven feedback loops for model improvement. This 2027 Summer internship provides hands-on experience applying ML knowledge to real-world LLM agent challenges.
Responsibilities
- Design evaluation systems for LLM-based agents covering task success, tool use, reasoning quality, and reliability.
- Build benchmarks and automated judging pipelines using rule-based checks, model-based judging, and human review.
- Analyze agent execution traces and user feedback to identify failure patterns and drive system improvements.
- Support the closed loop from experience to capability, collaborating with research, platform, and product teams to bring methods into production.
Requirements
- Currently pursuing a Bachelor's degree in Computer Science, AI, ML, Data Science, or related field.
- Solid foundation in machine learning and deep learning.
- Hands-on experience with LLM-based systems (agents, tool calling, retrieval, multi-agent systems) via research, internships, or projects.
- Strong Python skills and experience with a mainstream ML or agent evaluation framework.
- Demonstrated research or engineering ability through publications, substantial projects, internships, or open-source work.
Preferred Qualifications
- Publications at top-tier ML/NLP venues (NeurIPS, ICML, ICLR, ACL), especially in agent learning or evaluation.
- Experience with metric design, model-based judging, annotation, or statistical analysis.
- Familiarity with LLM post-training, reasoning/planning, continual learning, or large-scale log and trace analysis.