Every model faces the same fixed scenario pack, is scored on the same nine rubric axes, and is ranked by pairwise judgments from a single judge generation (—). Below: the scenarios, the criteria, the bias controls, and one important caveat about cost.
Every model plays out the same 25 short conversations — a friend venting after a rough day, a tense disagreement, someone testing a boundary — and a top-tier Claude model reads each one and rates it the way a thoughtful person would. Models are also compared head-to-head on the same conversation, and those wins become a chess-style Elo rating.
Same prompt for everyone. Each model gets the identical instructions (a warm, natural, spoken-voice companion — no lists, no markdown, no "as an AI") and the identical things said to it, word for word. No model gets a tailored prompt; the only thing that varies per model is the low-level chat formatting its software expects — the content is the same for all.
What this benchmark measures. OpenGauntlet is a test of Conversational Language Humanlikeness — how human a model reads in a written back-and-forth. It scores transcript-level qualities (word choice, warmth, emotional reasoning, flow), not spoken-voice qualities like prosody, intonation, or timing.
What "good" means here. The nine axes below reward genuine warmth, understanding the feeling behind the words, sounding like a person, staying in character, remembering what was said earlier, and healthy boundaries — while penalizing corporate- assistant tone and clichés. Engagement is judged through conversational flow (does it keep things alive and invite more, without being pushy?), and the voice metrics separately track how often replies end with a question. It rewards natural engagement, not just frequent engagement.
What the Brevity column is — and is not. It is a locally-measured composite of three things: the share of replies falling in a 20–80 word window, the share free of markdown, and type–token ratio. Measured against this corpus it tracks reply length almost exactly — a correlation of −0.93 between a model's mean reply length and its Brevity score. That is largely mechanical: the word-window term penalises long replies by construction, type–token ratio rises as texts get shorter regardless of quality (r = −0.78 here), and the markdown term varies by under 2.5 points across models, so it carries almost no signal. Read this column as “writes short, unadorned replies” — not as a verdict on prose quality. A model that writes long, descriptive prose will score low here by definition, and that is a description of its register, not a defect. It was previously labelled “Voice”, which invited exactly the wrong reading; the underlying measurement is unchanged.