Software Engineer

Role Overview

We are seeking experienced software engineers at the tech lead level to join our innovative project focused on building Large Language Model (LLM) evaluation and training datasets for realistic software engineering tasks. This role involves direct hands-on involvement in developing, automating, and maintaining complex software environments, as well as evaluating the quality and coverage of test cases in open-source repositories. The ideal candidate will possess a deep understanding of high-quality public GitHub repositories, with the ability to analyze, triage, and modify codebases effectively. You will work closely with researchers to identify challenging repositories and issues that can improve LLM performance in real-world coding scenarios.

Responsibilities

• Analyze and triage issues reported in trending open-source repositories on GitHub, identifying areas for improvement and testing.
• Set up and configure development environments, including Dockerization and environment management, to facilitate testing and evaluation.
• Assess unit test coverage and quality, providing insights to enhance repository robustness.
• Modify and run codebases locally to evaluate the performance of LLMs in bug-fixing and code comprehension tasks.
• Collaborate with AI researchers to design and identify repositories and issues that pose challenges for LLMs, contributing to dataset expansion and validation.
• Lead and mentor junior engineers, fostering a collaborative team environment and ensuring project milestones are met.
• Contribute to documentation, reporting, and continuous improvement of evaluation methodologies.

Requirements

• Minimum of 3+ years of professional experience in software engineering.
• Strong proficiency in at least one programming language such as Ruby.
• Hands-on experience with Git version control, Docker containerization, and setting up software pipelines.
• Ability to understand, navigate, and modify complex codebases efficiently.
• Experience in running, testing, and debugging real-world projects locally.
• Familiarity with open-source contributions, issue triaging, and project evaluation is advantageous.
• Excellent problem-solving skills and attention to detail.
• Strong communication skills and ability to collaborate effectively with cross-functional teams.