Contract Engineer

About the role

Evaluate post-trained MLLMs and build the datasets behind them.

Tasks

• Run and monitor evaluation suites across text, reasoning, tool use, multilingual, and multimodal tasks.
• Debug broken or inconsistent evaluations, and analyze model outputs to find and explain failure patterns.
• Build and curate evaluation and training datasets, and write concise reports on results.

Requirements

• Strong Python and data-analysis skills.
• Experience with LLM evaluation and benchmarks, including basic statistics such as confidence intervals and significance tests.
• Clear technical writing and careful attention to detail.
• Familiarity with LLM post-training (fine-tuning, reinforcement learning).
• Experience with tool-use, agent, or multimodal evaluations.
• Experience designing rubrics or grading criteria.
• Industrial or research experience is required (3+ years, with exceptions).
• This is NOT an internship position.

Schedule

• Contract

Pay

• The compensation is competitive ($200k+ per year).

How to apply

• Send your resume and a short description of relevant experience.
• The email subject should start with [LLM Contractor].

Location

• The team is in Seattle, but this position can be remote (in the US).