The novel contribution is a data-quality-first recipe for building a strong sub-1B-parameter multilingual text-embedding model (KaLM-Embedding) by adapting an autoregressive LLM (Qwen2-0.5B) instead of a BERT-style encoder, and by curating training data with three techniques: (1) persona-based synthetic data that distills diverse examples from LLMs; (2) ranking-consistency filtering that uses a reranker to drop uninformative and likely false-negative pairs; and (3) semi-homogeneous task-batch sampling that mixes same-task and random samples to balance in-batch-negative hardness against false-negative risk. Trained contrastively (weakly-supervised pre-training over 20+ data categories, then supervised fine-tuning over 70+), it sets a new state of the art among comparable-size models on the multilingual MTEB benchmark. The load-bearing claim is that cleaner, more diverse, well-batched training data - not architecture - is the primary lever for embedding quality, with weights and code released openly.
KaLM-Embedding is a general multilingual text-embedding model (technical report) whose central thesis is that training-data quality, more than architecture, drives embedding performance for retrieval…