Benchmarking LLMs on Journalistic Sourcing Annotation
A benchmark scenario for evaluating how well large language models identify and annotate sourcing in news stories: a five-category schema (adapted from journalism studies) covering sourced statements, source names and types, and source justifications; a hand-built ground-truth dataset of annotated articles; and an accuracy-scoring method that matches model output to ground truth via text embeddings and cosine-similarity thresholds. Findings show current LLMs struggle most at recovering every sourced statement and at spotting source justifications.
Measuring Large Language Models Capacity to Annotate Journalistic Sourcing
Since the launch of ChatGPT in late 2022, the capacities of Large Language Models and their evaluation have been in constant discussion and evaluation both in academic research and in the industry. S…