Efficient NLP
15 minutes
Zero-Shot Detectors Judge Writing by Its Own Predictability
One family of detectors needs no training examples at all. It runs the submitted text through a language model and scores two things: how predictable the words are, and how evenly that predictability is spread across the passage. Vendors call the second property burstiness, meaning the way human writing swings between plain sentences and odd ones while machine writing tends to hold a steady level. The strongest published version of this idea, DetectGPT, goes further: it perturbs the passage slightly and checks whether the model's probability drops sharply, which it does for machine-written text more than for human text. On news articles generated by a twenty-billion-parameter model, that method raised detection from 0.81 to 0.95 on the standard area-under-curve measure, in a controlled research setting with the generating model known. Classroom conditions are not that setting: the model is unknown, the text is short, and it has usually been edited. Detecting AI-generated images is a different problem with a different evidence base and none of these numbers transfer to it. You can now describe what a no-training detector actually measures and name two conditions that make its published score optimistic.