Separating Representation from Reconstruction Enables Scalable Text Encoders
Authors: Megi Dervishi, Mathurin Videau, Yann LeCun
Paper: https://arxiv.org/abs/2607.04011
Code: https://github.com/facebookresearch/lingua
Model: N/A
TL;DR
WHAT was done? The authors present CrossBERT, a novel bipartite text encoder architecture that decouples representation learning from token reconstruction. By utilizing a deep encoder for representation and a lightweight, cross-attention-only predictor for reconstruction, CrossBERT enables higher masking ratios (≥50%) and introduces a parallel Complementary Masking Strategy (CMS) to dramatically accelerate training and scale representation quality monotonically.
WHY it matters? Standard flat bidirectional encoders trained via Masked Language Modeling (MLM), such as BERT, suffer from severe representation degradation as they scale up, because their final layers over-specialize on reconstructing local vocabulary tokens rather than capturing high-level semantics. CrossBERT resolves this structural bottleneck, allowing text embeddings to scale predictably with compute and enabling frozen representation features that can match or outperform fully fine-tuned baselines in retrieval and semantic tasks.
Details
The Representation Collapse of Flat Bidirectional Backbones
While decoder-only architectures have scaled predictably and dominated generative modeling, bidirectional encoders (more on different architectures here) have stagnated since the introduction of BERT. Recent modernization efforts, such as ModernBERT [review] and NeoBERT, have focused primarily on dataset scaling and post-training finetuning, keeping the underlying flat architecture unchanged. The authors expose a critical failure mode of this flat design: when evaluated under frozen probing conditions, standard MLM models show a dramatic degradation in representation quality as parameter count and pre-training compute increase. This phenomenon, visualized in Figure 1, shows that while larger models achieve better pre-training perplexity, their final layer representations become increasingly unexploitable by frozen linear and kNN probes on standard benchmarks like GLUE and MTEB. This failure stems from a fundamental misalignment in the flat BERT architecture, which forces the identical set of parameters to simultaneously extract rich semantic abstractions and perform local token-level reconstruction.
