Assembling a Small Labelled Eval Set Before a Prompt Reaches Production

the unit of professional work is the eval set, not the prompt; without one every later judgement is anecdote

jeremy Video

How to Systematically Setup LLM Evals Using Unit Tests and Human-in-the-Loop Metrics

LLM evaluation is defined as a systematic measurement framework comprising three hierarchical levels: Unit Tests (Level 1), Human-in-the-Loop and Model-as-Judge evaluations (Level 2), and A/B Testing…