Transport and Merge: Cross-Architecture Merging for Large Language Models
Use activation correspondences to transfer knowledge between language models whose architectures do not match.
The short version
Transport and Merge uses optimal transport over activation relationships to establish correspondences between heterogeneous language models. These correspondences guide layer and feature alignment, making cross-architecture weight fusion possible. A small shared input set supplies the evidence needed to connect the two models.
The work evaluates knowledge transfer in low-resource language and domain settings, and explores residual-frozen adaptation after fusion. It separates the problem of finding a functional correspondence from the subsequent problem of integrating the transferred representation.
Transferring knowledge across architectures begins with finding which internal computations correspond.
A transport map between architectures
Activation correspondences guide layer transport, feature transport, and weight fusion.
View in the paperWhy this matters
Conventional merging often assumes compatible architectures. That assumption limits transfer from high-resource source models to heterogeneous targets with limited task or language data.
How it works
- Collect activations from source and target models on a small set of shared inputs.
- Solve optimal transport to establish correspondences between heterogeneous neuronal representations.
- Use the resulting transport plans to guide direct weight-space fusion.
How concentrated are the correspondences?
The plot measures the transport mass explained by the leading neuron correspondences.
View in the paperWhat the study finds
Experiments across low-resource languages and specialized domains report consistent improvements over the corresponding target models. The transfer uses a small input set to infer the cross-model correspondence.
Transfer in a low-resource language
The original MalayMMLU table compares the base model with fusion and subsequent adaptation.
View in the paperWhat this means
This frames model merging as a problem of translation between internal coordinate systems. Architecture can obscure correspondence even when two models contain related computations; activation relationships provide an empirical bridge across that boundary.
The useful outcome is reusable knowledge under limited adaptation data. Future progress depends on identifying when transport plans preserve function, how shared-input coverage affects the mapping, and whether gains survive beyond the domains that supplied the correspondence evidence.
Paper and resources
This is an introduction to the linked paper. Refer to the original for method definitions, evaluation settings, and complete results.
Cite this work
@misc{transport_and_merge,
title = {Transport and Merge: Cross-Architecture Merging for Large Language Models},
author = {Cui, Chenhang and Yang, Binyun and Shen, Fei and Chen, Yuxin and Zheng, Jingnan and Wang, Xiang and Zhang, An and Chua, Tat-Seng},
year = {2026},
eprint = {2602.05495},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2602.05495}
}