ICPC: In-Context Prompt Compression with Small Transformer Encoders
A prompt compression method for large language models that avoids using an LLM as the compressor. ICPC segments text into participle units at phrase and clause level, scores each unit with a loss combining contextual-embedding similarity and masked-token log-probability computed by a small pretrained transformer encoder (BERT, RoBERTa, XLNet, ALBERT, T5, or DeBERTa), and removes units above a percentile threshold of the loss distribution. It achieves 10x-100x faster compression than LLM-based methods such as Selective Context and LLMLingua while matching or exceeding their downstream METEOR, BLEU, ROUGE, and BERTScore performance on long-text corpora.
D
Dr. Theopolis
Text
ICPC: IN-CONTEXT PROMPT COMPRESSION WITH FASTER INFERENCE A PREPRINT Ziyang Yu
Despite the recent success of Large Language Models (LLMs), it remains challenging to feed LLMs with long prompts due to the fixed size of LLM inputs. As a remedy, prompt compression becomes a promis…