Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data
Authors: Shikai Qiu, Marc Finzi, Yujia Zheng, Kun Zhang, Andrew Gordon Wilson
Paper: https://arxiv.org/abs/2607.11883
Code: https://github.com/shikaiqiu/requential-coding
Model: N/A
TL;DR
WHAT was done? The authors introduce requential coding, a lossy model compression framework that compresses a generative model by encoding its training trajectory rather than its raw parameters. In this setup, a student model is trained iteratively on synthetic data it generates itself, with training samples selected by a stronger teacher model. By utilizing relative entropy coding, the transmission cost is limited solely to the student-teacher disagreements, bypassing the parameter count and data entropy bounds that inflate traditional compression schemes.
WHY it matters? This framework resolves a long-standing tension in deep learning: why massive, overparameterized models generalize exceptionally well despite their huge capacity. Requential coding achieves code lengths orders of magnitude shorter than parameter-based quantization or traditional prequential coding, leading to state-of-the-art, non-vacuous PAC-Bayes generalization guarantees for billion-parameter LLMs. Crucially, the certified generalization gap tightens as models scale in the compute-optimal regime, aligning deep learning scaling laws with classical Occam’s razor principles.
Details
The Parameter-Data Complexity Bottleneck
A central paradox of modern deep learning is that overparameterized neural networks generalize remarkably well, yet showcasing this simplicity through model compression remains highly challenging. Traditional parameter-based compression methods, such as post-training quantization like GPTQ or QuIP#, yield code lengths that scale linearly with the model’s parameter count, regardless of how much actual information those parameters represent. Conversely, data-centric compression schemes like Prequential Coding bypass parameter counts by compressing the training data itself via the model’s sequential predictions. However, prequential coding suffers from a separate bottleneck: it must losslessly encode the exact training sequence. This forces it to pay a heavy tax on the irreducible entropy of the data source, even after the model has extracted all learnable regularities.
To overcome these dual limitations, a compressor must discard unimportant microscopic details of both the parameters and the data. The delta of the proposed method lies in compressing an approximate model trained on a surrogate dataset that is vastly cheaper to describe. By transmitting only the specific ways in which the training data departs from what the model already knows, the complexity of the description is decoupled from both the total parameter count and the irreducible data entropy.
