NExT-LFM · Research overview
More research
Fundamental Research / Correctness evaluation / Research overview

Circe: Training-Free LLM Self-Evaluation via Component Interaction Consistency

Read cross-layer attention–FFN interaction consistency, representation geometry, and output likelihood to assess a response without reference answers or repeated sampling.

Miao Shuyi

Preprint 4-minute read Read the paper PDF
Concept illustration: Circe: Training-Free LLM Self-Evaluation via Component Interaction Consistency
Concept illustration inspired by the study.

The short version

A model can confidently repeat an incorrect answer. Circe asks whether the computation behind that answer offers evidence beyond the probability assigned to its words. It measures how attention updates in earlier layers align with attention-induced FFN changes in later layers, then combines that interaction signal with representation geometry and output likelihood.

The evaluator needs one generated response and no additional model training. Its fusion weights are selected on development data; reference answers are not required when scoring a new response. The distinction matters: training-free self-evaluation still needs its calibration and evaluation protocol to be stated clearly.

71.76%Mean AUROC in the component ablation
5 LLMsSix benchmarks across four domains
1 responseNo repeated sampling

Correctness is worth examining in the interaction between components, as well as in the confidence of the final answer.

Figure 2 · Three complementary sources of correctness evidence. The original framework separates attention–FFN interaction consistency, hidden-state geometry, and likelihood confidence before fusion.
Figure 2 / p. 3

Three complementary sources of correctness evidence

The original framework separates attention–FFN interaction consistency, hidden-state geometry, and likelihood confidence before fusion.

View in the paper

Why this matters

Hidden-state trajectories aggregate the effects of multiple components. Two answers can have similar output confidence while differing in how attention and feed-forward updates interact. Studying those interactions provides a more specific diagnostic target than aggregate representational drift alone.

How it works

  1. Measure component interaction consistency (CIC): compare earlier attention-update directions with the FFN-output changes induced by attention at later layers.
  2. Compute representation geometry (RG) from normalized displacement and angular changes, and likelihood confidence (LC) from the generated tokens’ mean negative log-likelihood.
  3. Fuse the internal and external signals into a correctness score using development-selected weights, then rank responses without repeated sampling or a reference answer at inference.
Figure 3 · Correct and incorrect responses have different score distributions. On Qwen2-7B-Instruct and MATH, the three signals have higher means for correct responses, with remaining overlap between the classes.
Figure 3 / p. 4

Correct and incorrect responses have different score distributions

On Qwen2-7B-Instruct and MATH, the three signals have higher means for correct responses, with remaining overlap between the classes.

View in the paper

What the study finds

The paper evaluates five LLMs on six benchmarks across mathematics, reasoning, knowledge, and understanding. The complete method has the best reported domain-average AUROC, FPR95, and AUPR, and leads AUROC in 18 of 20 model–domain pairs.

In the component ablation, mean AUROC is 71.76%; removing CIC reduces it to 69.71%. Additional evaluations cover 11 languages and a visual geometry task, where the reported AUROC is 71.25%. These are ranking results for response correctness, not task-answering accuracy.

Table 2 · Interaction consistency contributes beyond geometry and likelihood. The full score reaches 71.76% mean AUROC across the five models; removing CIC gives 69.71%.
Table 2 / p. 6

Interaction consistency contributes beyond geometry and likelihood

The full score reaches 71.76% mean AUROC across the five models; removing CIC gives 69.71%.

View in the paper

What this means

The central insight is that reliability can depend on how a model coordinates its components, not just on how strongly it prefers its output. Interaction consistency offers a way to inspect that coordination and contributes information that trajectory geometry and likelihood leave out.

This supports computational evidence as an additional axis of model evaluation. Consistency is still a statistical signal rather than a certificate of truth: its usefulness must be checked against task, model, and calibration changes.

Paper and resources

Contact the authorMiao Shuyi
shuyimiao@buaa.edu.cn

This is an introduction to the linked paper. Refer to the original for method definitions, evaluation settings, and complete results.

Cite this work
@misc{circe,
  title = {Circe: Training-Free LLM Self-Evaluation via Component Interaction Consistency},
  author = {Miao, Shuyi and Li, Zheng and Xie, Xu and Xu, Linrui and Shen, Fei and Du, Xiaoyu and Chua, Tat-Seng},
  year = {2026},
  eprint = {2611.07948},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07948}
}