CaseSumm is the largest open dataset for long-context legal case summarization: 25.6K U.S. Supreme Court opinions paired with their official 'syllabus' summaries, extending back to 1815. Alongside the corpus, its authors report that automatic metrics (e.g. ROUGE) and LLM-based evaluators diverge sharply from expert human judgement — a smaller model can top automatic metrics while human annotators flag its summaries as hallucinated (precedent-citation errors, misrepresented facts), showing why human evaluation remains essential for high-stakes legal summarization.
This paper introduces CaseSumm, a novel dataset for long-context summarization in the legal domain that addresses the need for longer and more complex datasets for summarization evaluation. We collec…