Factual Knowledge Assessment of Language Models Using Distractors

Language models encode extensive factual knowledge within their parameters. The accurate assessment of this knowledge is crucial for understanding and improving these models. In the literature, factual knowledge assessment often relies on cloze sentences, which can lead to erroneous conclusions, due to the complexity of natural language (out-of-subject continuations, the existence of many correct answers, and the several ways of expressing them). Moreover, current evaluation methods rely heavily on synthetic updates, they are restricted to a single type of update scenario, and often overestimate the updating success. To overcome these issues, this project relies on WikiFactDiff, a dataset of real-world factual changes automatically extracted, by comparing Wikidata snapshots across time, and integrates it with a distractor-based assessment for precise evaluation.

Our method was published at COLING 2025.

The code for reproducing the experiments is available on GitHub, under the MIT licence.