Evaluation of RAG Metrics in an Application Context

Nowadays, the evaluation of RAG (Retrieval Augmented Generation) systems relies heavily on LLM-as-a-judge metrics. However, the reliability of these metrics varies depending on the application context. It is therefore useful, at the beginning of the development of a RAG system, to evaluate the available metrics in order to verify, that they provide acceptable approximations of the considered criteria, and that they will enable reliable comparisons of the different versions of the developed RAG system. Generally, metrics are evaluated by measuring their correlation with scores given by humans.

The paper Evaluation of RAG Metrics in an Application Context: an Experiment, its Findings and its Limitations, published at EvalLLM 2026, reports an experiment aimed at evaluating RAG metrics in the context of assessing an internal RAG solution on business data.

The repository, under the MIT and CC-BY licences, contains the code and data necessary to reproduce the analyses, starting from the scores generated by the various metrics, and by the human evaluators.