Browse Teach Curate

Deterministic Vector-Database Scoring for Evaluating Open-Ended LLM Reasoning

An LLM-evaluation method that stores human-graded reference answers as embeddings in a vector database, retrieves the closest reference to a model's open-ended answer by cosine similarity, and uses a small local LLM to deterministically score factual correctness, reducing fluency bias and human-grading reliance versus multiple-choice benchmarks.

Data Text

2501.00257

This paper introduces EQUATOR (Evaluation of Question Answering Thoroughness in Open-ended Reasoning), a deterministic framework for evaluating the factual accuracy and reasoning of LLMs on open-ende…