- Client
- Multiple frontier AI labs
- Role
- AI Training Specialist / Evaluator
- Year
- 2024
- Platform
- Web-based evaluation tools
Fig. 01 is an illustrative schematic of the evaluation loop. No client material is shown.
The problem
Frontier AI labs need structured human ratings to train and evaluate their models.
AI training specialist and evaluator for multiple frontier labs in 2024: adversarial prompts, rubrics, golden responses and reasoning-trace evaluation across text, audio and multi-turn systems.
My ratings are included in the training data the frontier labs use for the upcoming releases of their flagship models.
Human evaluation
Rating each quality on its own
Fig. 02 · Rubric constellation
Choosing the golden response from eight
Each axis is rated on its own, so a fluent answer can’t hide a factual miss. Each rating follows the rubric, not a general impression of the answer.
R-06 · Golden responseHolds up on every axis.
Illustrative. Rebuilt in code from Fig. 02 of the original case study; the bars read each shape’s reach on an axis, with the golden response marked in gold. No client data.
Flagship assistant evaluation
Separate fluency from factual reliability.
For a new version of a major AI lab’s flagship conversational assistant, I evaluated conversational quality, factual reliability and alignment with user intent as separate dimensions.
That distinction matters when an answer sounds helpful but mishandles the evidence or the request.
Assistant and copilot evaluation
What I checked in every response.
- 01
Complex instructions
Did the model follow them, explicit and implicit?
- 02
Grounded, accurate answers
Is every claim supported by its source, and true?
- 03
Safety guardrails
Did the response stay within them?
Reviewing the reviewers
I also reviewed other evaluators’ ratings for consistency, accuracy and adherence to the rubric. This work keeps the model training data pipeline accurate and contributes to model safety and model alignment.
Credits
- Role
- AI Training Specialist / Evaluator
- Work
- Multi-axis LLM evaluation, SFT and RLHF rating, rubric interpretation, cross-model evaluation, quality assurance
- Worked with
- Multiple frontier AI labs, alongside QA and evaluator teams
- Modalities
- Text, audio and multi-turn systems
- Tools
- Custom evaluation tools and web-based rating platforms
- Year
- 2024