Adam Rotmil
Frontier LLM evaluation · RLHF and SFT

Client
Multiple frontier AI labs
Role
AI Training Specialist / Evaluator
Year
2024
Platform
Web-based evaluation tools
Inside the work

Fig. 01 is an illustrative schematic of the evaluation loop. No client material is shown.

The problem

Frontier AI labs need structured human ratings to train and evaluate their models.

My role

AI training specialist and evaluator for multiple frontier labs in 2024: adversarial prompts, rubrics, golden responses and reasoning-trace evaluation across text, audio and multi-turn systems.

Outcome

My ratings are included in the training data the frontier labs use for the upcoming releases of their flagship models.

Human evaluation

Rating each quality on its own

Fig. 02 · Rubric constellation

Choosing the golden response from eight

Each axis is rated on its own, so a fluent answer can’t hide a factual miss. Each rating follows the rubric, not a general impression of the answer.

R-06 · Golden responseHolds up on every axis.

Illustrative. Rebuilt in code from Fig. 02 of the original case study; the bars read each shape’s reach on an axis, with the golden response marked in gold. No client data.

Flagship assistant evaluation

Separate fluency from factual reliability.

For a new version of a major AI lab’s flagship conversational assistant, I evaluated conversational quality, factual reliability and alignment with user intent as separate dimensions.

That distinction matters when an answer sounds helpful but mishandles the evidence or the request.

Assistant and copilot evaluation

What I checked in every response.

Assistant and copilot products · Multiple quality axes

  1. 01

    Complex instructions

    Did the model follow them, explicit and implicit?

  2. 02

    Grounded, accurate answers

    Is every claim supported by its source, and true?

  3. 03

    Safety guardrails

    Did the response stay within them?

Reviewing the reviewers

I also reviewed other evaluators’ ratings for consistency, accuracy and adherence to the rubric. This work keeps the model training data pipeline accurate and contributes to model safety and model alignment.

Credits

Role
AI Training Specialist / Evaluator
Work
Multi-axis LLM evaluation, SFT and RLHF rating, rubric interpretation, cross-model evaluation, quality assurance
Worked with
Multiple frontier AI labs, alongside QA and evaluator teams
Modalities
Text, audio and multi-turn systems
Tools
Custom evaluation tools and web-based rating platforms
Year
2024
NextWild BrainsShaping a brain wearable, from first research to the pitch that funded it