Circe: Training-Free LLM Self-Evaluation via Component Interaction Consistency
Read cross-layer attention–FFN interaction consistency, representation geometry, and output likelihood to assess a response without reference answers or repeated sampling.
The short version
A model can confidently repeat an incorrect answer. Circe asks whether the computation behind that answer offers evidence beyond the probability assigned to its words. It measures how attention updates in earlier layers align with attention-induced FFN changes in later layers, then combines that interaction signal with representation geometry and output likelihood.
The evaluator needs one generated response and no additional model training. Its fusion weights are selected on development data; reference answers are not required when scoring a new response. The distinction matters: training-free self-evaluation still needs its calibration and evaluation protocol to be stated clearly.
Correctness is worth examining in the interaction between components, as well as in the confidence of the final answer.
Three complementary sources of correctness evidence
The original framework separates attention–FFN interaction consistency, hidden-state geometry, and likelihood confidence before fusion.
View in the paperWhy this matters
Hidden-state trajectories aggregate the effects of multiple components. Two answers can have similar output confidence while differing in how attention and feed-forward updates interact. Studying those interactions provides a more specific diagnostic target than aggregate representational drift alone.
How it works
- Measure component interaction consistency (CIC): compare earlier attention-update directions with the FFN-output changes induced by attention at later layers.
- Compute representation geometry (RG) from normalized displacement and angular changes, and likelihood confidence (LC) from the generated tokens’ mean negative log-likelihood.
- Fuse the internal and external signals into a correctness score using development-selected weights, then rank responses without repeated sampling or a reference answer at inference.
Correct and incorrect responses have different score distributions
On Qwen2-7B-Instruct and MATH, the three signals have higher means for correct responses, with remaining overlap between the classes.
View in the paperWhat the study finds
The paper evaluates five LLMs on six benchmarks across mathematics, reasoning, knowledge, and understanding. The complete method has the best reported domain-average AUROC, FPR95, and AUPR, and leads AUROC in 18 of 20 model–domain pairs.
In the component ablation, mean AUROC is 71.76%; removing CIC reduces it to 69.71%. Additional evaluations cover 11 languages and a visual geometry task, where the reported AUROC is 71.25%. These are ranking results for response correctness, not task-answering accuracy.
Interaction consistency contributes beyond geometry and likelihood
The full score reaches 71.76% mean AUROC across the five models; removing CIC gives 69.71%.
View in the paperWhat this means
The central insight is that reliability can depend on how a model coordinates its components, not just on how strongly it prefers its output. Interaction consistency offers a way to inspect that coordination and contributes information that trajectory geometry and likelihood leave out.
This supports computational evidence as an additional axis of model evaluation. Consistency is still a statistical signal rather than a certificate of truth: its usefulness must be checked against task, model, and calibration changes.
Paper and resources
This is an introduction to the linked paper. Refer to the original for method definitions, evaluation settings, and complete results.
Cite this work
@misc{circe,
title = {Circe: Training-Free LLM Self-Evaluation via Component Interaction Consistency},
author = {Miao, Shuyi and Li, Zheng and Xie, Xu and Xu, Linrui and Shen, Fei and Du, Xiaoyu and Chua, Tat-Seng},
year = {2026},
eprint = {2611.07948},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2611.07948}
}