Scale AI logo for Machine Learning Research Scientist, Evaluations job on DevsUnite

Machine Learning Research Scientist, Evaluations

Scale AI

Actively HiringFull-time
Apply
Location

San Francisco, CA; Seattle, WA; New York, NY

Experience

2-4 years

Role

Machine Learning Research Scientist, Evaluations

Salary

Competitive

Share this job

Know someone who'd be a great fit?

Job Description

Role Overview

Scale partners with leading AI labs to advance generative AI research, seeking Research Scientists and Engineers expert in large language model post-training (SFT, RLHF, reward modeling) to build benchmarks and diagnose model failure modes in text and multimodal settings.

Responsibilities

  • Analyze model behavior to identify, characterize, and diagnose failure modes in frontier LLMs and Agents, focusing on root cause analysis (RCA) of capability gaps, reasoning errors, robustness, and alignment issues.
  • Design and build benchmarks and evaluation methods that measure LLM capabilities in both text and multimodal modalities.
  • Apply post-training expertise (SFT, RLHF, reward modeling) to connect observed failures to the data and training interventions that address them.
  • Publish research findings in top-tier AI conferences.

Requirements

  • Ph.D. or Master's degree in Computer Science, Machine Learning, AI, or a related field.
  • Deep understanding of deep learning, reinforcement learning, and large-scale model fine-tuning.
  • Experience with post-training techniques such as RLHF, preference modeling, or instruction tuning, and with LLM evaluation or benchmark development.
  • Excellent written and verbal communication skills.
  • Published research in areas of machine learning at major conferences (NeurIPS, ICML, ICLR, ACL, EMNLP, CVPR, etc.) and/or journals.
  • Previous experience in a customer-facing role.

Required Skills

Machine LearningLLMsGenerative AIDeep LearningRLHF

Ready to apply for Machine Learning Research Scientist, Evaluations?

See how you stack up at Scale AI before you apply, or go straight to the application.

Apply Anyway