CodeElo: Competition-Level LLM Code Benchmark with Human-Comparable Elo Ratings
A benchmark that measures large language models' competition-level coding ability by compiling six months of Codeforces contest problems (with division, difficulty, and algorithm tags) and judging model solutions through direct submission to Codeforces - reusing the platform's private test cases, special judges, and exact execution environment. Each model receives an Elo rating computed like human contestants' but with lower variance, yielding a leaderboard of 33 LLMs directly comparable to human percentiles, plus analyses of per-algorithm performance and C++ versus Python.
CODEELO: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings
A benchmark for measuring the competition-level code-generation ability of large language models. CodeElo is built from the last six months of Codeforces contest problems (tagged with division, diffi…