This work uses state-of-the-art large language models (Gemini and OpenAI models) together with medical ontologies (RxNorm and SNOMED) as automatic 'teacher labelers' to annotate clinical notes for medications, diseases, and symptoms, then distills that supervision into BioBERT student models roughly a thousand times smaller. Across more than 3,300 notes from five public datasets plus external validation on MedAlign, the distilled BioBERT models reach F1 scores close to their LLM teachers and to BERT fine-tuned on human labels (e.g. 0.87 vs 0.84 teacher for medication), while running up to 12x faster and up to 101x cheaper than the LLMs. The contribution is a scalable, low-cost alternative to running large LLMs for clinical information extraction: ontology-grounded LLM teacher labeling feeding knowledge distillation into deployable small encoders.
Large language models (LLMs) excel at clinical information extraction but their computational demands limit practical deployment. Knowledge distillation--the process of transferring knowledge from la…