How to benchmark an LLM agent that plays the clinician in psychiatric assessment interviews without exposing real patients: simulate the patient instead, from an explicit multi-faceted construct specifying profile, history, and behavior, then score the assessing agent on how accurately it recovers that construct from the conversation. Covers construct-grounded utterance simulation, rubric-weighted comparison of the agent's predicted construct against the simulated patient's ground truth to yield a quantitative score, validation by correlating automated scores with expert psychiatrist judgments and inter-rater reliability statistics, and safety checks including jailbreak testing. The design goals are clinical relevance, ethical safety, cost efficiency, and quantitative comparability.
PSYCHE is an evaluation framework for LLM-based psychiatric assessment conversational agents (PACAs), the systems meant to play the psychiatrist's role in clinical interviews. It simulates psychiatri…