SuperAI: Merging Capabilities Across Heterogeneous Foundation Models
Align functional subspaces across heterogeneous foundation models through orthogonal rotation, bringing complementary capabilities from multiple sources into one host model.
The short version
SuperAI studies how a host model can incorporate capabilities from heterogeneous source models. It identifies corresponding functional subspaces, aligns their coordinates, and learns residual mappings into the host. The useful interface is a relationship between representations, allowing sources with different architectures to contribute without requiring direct parameter correspondence.
The framework combines orthogonal subspace alignment, layer selection, and joint training of lightweight mappings. Experiments with multiple hosts and domains examine both specialist gains and retention of the host’s general capabilities as the number of sources grows.
Capabilities become composable when their representations can be aligned across model boundaries.
A representation interface for composition
Subspace alignment, layer selection, feature mapping, and joint training connect source expertise to a host model.
View in the paperWhy this matters
Useful capabilities are distributed across models with incompatible architectures and representations. Their coordinates must be aligned before source knowledge can be meaningfully injected into a host.
How it works
- Use shared anchor samples and PCA to construct source and host subspaces with matching dimensions.
- Align source and host coordinates with orthogonal Procrustes mappings.
- Train residual MLP mappings with both task outputs and source-model increments as supervision.
Before and after alignment
Source and host representations are compared in alignment-sensitive projections.
View in the paperWhat the study finds
Experiments cover three host models and 15 capability domains. The framework reports higher average domain scores than the evaluated cross-architecture alternatives and better retention of general capabilities as more source models are added.
Retaining the host’s general capability
The comparison tracks MMLU retention as the number of source experts increases.
View in the paperWhat this means
The conceptual shift is to treat a trained model as a possible source of reusable internal computation. Architecture still matters, but functional compatibility offers another organizing principle for combining expertise. Representation alignment becomes a practical question about which information can be carried into a host and used there.
A useful composition system must also preserve the host’s existing competence and manage interference among sources. The reported evaluations make those tradeoffs visible; extending this approach to broader tasks will require testing whether alignment quality continues to predict usable capability transfer.
Paper and resources
This is an introduction to the linked paper. Refer to the original for method definitions, evaluation settings, and complete results.
Cite this work
@misc{superai,
title = {SuperAI: Merging Capabilities Across Heterogeneous Foundation Models},
author = {Dou, Jingtong and Shi, Chuancheng and Wu, Wenhua and Bao, Wei and Shen, Fei and Wang, Zhiyong and Chua, Tat-Seng},
year = {2026},
eprint = {2610.15322},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.15322}
}