Study reveals AI models ignore financial risk disclosures despite reading them accurately

A research paper by Miao Liu and Chi-Chi Liu, titled "Reading Is Not Using: Retrieval, Judgment, and Pipeline Design in AI Finance Research", reveals a fundamental flaw in deploying large language models as financial analysts to support investment decisions. Financial institutions have routinely evaluated these models based on their raw ability to retrieve information from texts, yet the experimental findings demonstrate a sharp gap between retrieving information and actually integrating it into the final judgment.

The researchers conducted controlled experiments by holding the target company's financial data constant while gradually increasing irrelevant context from 2,000 tokens to 128,000 tokens. The results showed thatthe impact of risk disclosures on investment decisions diminishesto the level of empirical noise as context expands, even though the model's ability to retrieve the direct disclosure text remains entirely accurate. This pattern recurred across different language model families and multiple evaluation tasks, and was confirmed in experiments where genuine disclosures were removed from actual annual filings.

The paper demonstrates that more capable models merely delay the emergence of this gap rather than eliminate it, meaning that increasing model capability alone is not a sufficient solution. Through causal interventions on memory, the researchers proved that carrying financial disclosures through to the final decision requires an integration of dense summaries and direct searches within the source text. Pipeline architecture emerges here as a decisive factor, as workflows based on text chunking and summarization obscure essential information, whereas targeted structural reformulation placed directly beside the decision point successfully restores the lost impact of disclosures.

This finding carries direct implications for financial analysis, investment, and risk management teams at banks and investment funds across the Gulf, Egypt, and the Levant, which are racing to build AI pipelines to process annual disclosures and prepare investment memos. Relying on conventional retrieval benchmarks provides false confidence in systems that ignore risk data during actual financial decision-making. This requires engineering teams to redesign pipeline architectures to anchor structured disclosures next to the decision window, rather than merely expanding the context window or replacing the model with a newer release.

The study concludes thatthe performance of an AI financial analyst is determined by coupling model capability with pipeline architecturetogether, and that evaluation metrics based solely on retrieval may validate and approve systems that completely ignore information they have proven capable of retrieving verbatim. Restructuring analysis pipelines thus becomes an essential technical requirement for any investment system relying on AI to evaluate risks and allocate capital.