OrthoEraser: Coupled-Neuron Orthogonal Projection for Concept Erasure
Erase a sensitive concept while explicitly protecting the benign features coupled to the intervention.
The short version
OrthoEraser studies concept erasure through the geometry of entangled features. It locates sensitive neurons and identifies coupled features whose responses overlap with the target. An orthogonal projection constrains the intervention relative to the coupled benign subspace, aiming to suppress the target while reducing collateral changes.
The evaluation pairs removal effectiveness with generation fidelity and feature-level analysis. This makes selectivity a first-class objective: a stronger intervention is useful when its behavioral effect is specific enough to preserve legitimate generation.
Precise concept removal requires understanding what the target shares with the capabilities we want to preserve.
Locating sensitive layers and neurons
Layer-level sensitivity and neuron-level scores narrow the target of intervention.
View in the paperWhy this matters
Suppressing a whole unit can also remove benign attributes because sensitive and benign semantics need not occupy orthogonal activation spaces.
How it works
- Decompose dense activations with an SAE and identify sensitive units.
- Detect coupled benign units that are vulnerable to the proposed intervention.
- Project erasure vectors into the coupled units’ null space using analytical orthogonalization.
Inspecting feature entanglement
Activation shifts are compared across sensitive, coupled, and other neurons for two concept groups.
View in the paperWhat the study finds
The paper evaluates erasure precision and preservation of non-sensitive generation together. Reported comparisons support the benefit of explicitly protecting the identified benign subspace.
Suppression and preservation
Intervention strategies are compared on target detection, image-text alignment, and generation fidelity.
View in the paperWhat this means
Controllability depends on the resolution of an intervention. When harmful and benign features share directions, indiscriminate suppression can damage both. Geometry provides a way to express which components should change and which should be protected.
The broader lesson is to evaluate editing through a joint account of suppression and preservation. The tested projection offers evidence for a more selective mechanism; its guarantees are limited to the modeled subspace and experimental settings, motivating continued checks for residual behavior and side effects.
Paper and resources
This is an introduction to the linked paper. Refer to the original for method definitions, evaluation settings, and complete results.
Cite this work
@misc{orthoeraser,
title = {OrthoEraser: Coupled-Neuron Orthogonal Projection for Concept Erasure},
author = {Shi, Chuancheng and Wu, Wenhua and Shen, Fei and Zhu, Xiaogang and Hu, Kun and Wang, Zhiyong},
year = {2026},
eprint = {2603.11493},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2603.11493}
}