Software Engineer - Evaluation (Remote)

Role Overview

We are seeking a Senior Software Engineer – Go (LLM Evaluation & Repository Validation) for a part-time remote role. The project involves building LLM evaluation and training datasets to train models on realistic software engineering problems by developing verifiable tasks based on public repository histories using a synthetic approach with human-in-the-loop input.

Responsibilities

• Analyze and triage GitHub issues across trending open-source libraries to assess LLM performance.
• Set up and configure code repositories, including Dockerization and environment setup for testing.
• Evaluate unit test coverage and quality across complex codebases.
• Modify and run codebases locally to assess how LLMs perform in bug-fixing scenarios.
• Collaborate with researchers to design and identify repositories and issues that challenge LLMs.

Requirements

• Minimum 3+ years of overall software engineering experience.
• Strong proficiency in Go and familiarity with Git, Docker, and basic software pipeline setup.
• Ability to understand and navigate complex codebases efficiently.
• Experience running, modifying, and testing real-world projects locally.
• Comfortable contributing to or evaluating open-source projects.

Skills

• Go
• Docker
• Git
• GitHub
• LLM Evaluation