IAR: A three-stage framework that integrates document knowledge into language model weights without retrieval

Large language models encounter notable failures when answering questions tied to specific, limited sets of documents if those source documents are not retrieved during the inference and model execution phase. To address this technical issue, a new research paper examines this setup under the title document knowledge assimilation, meaning converting a fixed, specific collection of texts into internal parametric knowledge that can be used directly to answer questions without external retrieval systems. For this purpose, the researchers propose a post-training framework divided into three consecutive stages called "IAR", an acronym that combines injection, alignment, and restoration.

The framework relies on a systematic separation of three core operations: organized document knowledge injection, answer behavior alignment, and restoration of the model’s general capabilities.While traditional continual pretraining relies on generic methods, the injection stage devises a different pathway that transforms source documents into composite training targets that include text completion, paraphrasing, and conditional reconstruction following instructions. This is followed by the alignment stage, which adapts the knowledge-injected model through dedicated supervision focused solely on answers to fine-tune the response-generation behavior to questions. In the third stage, restoration is performed by merging the specially fine-tuned model with the base instruction model to recover comprehensive capabilities and protect them from degradation.

The practical experiments evaluating the new methodology extended across two datasets, the Common Corpus (CC) and CCI, and were applied on four prominent language model families: Llama, Phi, Qwen, and SmolLM. Experimental results showed that the IAR framework expands and improves the gap between specialized-domain performance and general linguistic performance when assimilating documents without retrieval. In the primary comparison, the proposed framework outperformed traditional fine-grained supervised training, known as Vanilla SFT, on all four recorded metrics, across seven of eight configurations combining models and datasets.

The experiments recorded clear gains in specialized knowledge assimilation accuracy while preserving the overall model balance.The average gains achieved were 3.6 percentage points in accuracy on specialized-domain questions, alongside an average increase of 12.1 percentage points in overall performance on IFEval, MMLU, and MSBench tests. When expanding the baseline comparison on Common Corpus data, the results showed that methods such as LoRA and FAPM may record superiority on isolated general metrics, yet the IAR framework uniquely offers one of the strongest overall evaluations among all approaches that have reached advanced or near-leading levels in document knowledge assimilation. These results appear in a research paper led by Hua Zhu, with equal contributions from Qian Ke and Xiaofeng Shi as co-authors, together with researcher Xiaosong Qiu.