Small LLMs: Pruning vs. Training from Scratch
Authors: Yufeng Xu, Taiming Lu, Kunjun Li, Jiachen Zhu, Mingjie Sun, Zhuang Liu
Paper: https://arxiv.org/abs/2606.14150
Code: https://github.com/zlab-princeton/pruning-vs-scratch
Model: https://ai.meta.com/blog/meta-llama-3-1
TL;DR
WHAT was done? The authors perform a controlled, token-matched empirical evaluation comparing network pruning against training from scratch for small Large Language Models (LLMs). By evaluating six pruning methods across five granularities (depth, width, depth-and-width, 2:4 semi-structured, and unstructured sparsity), the study isolates the initialization value of pruned weights under two distinct settings: equal target retraining token budgets and equal total pipeline token budgets.
WHY it matters? This work reframes network pruning from a pure post-hoc compression technique into a token-efficient initialization strategy. Crucially, it demonstrates that when total compute and token budgets are equalized across the entire pipeline, coarse structured pruning behaves primarily as neural architecture search—yielding models that can be matched or surpassed by training a smaller dense architecture from scratch. Conversely, fine-grained sparse pruning transfers specific weight-level knowledge that additional training tokens cannot recover.
Executive summary: For research leaders and engineering managers navigating model deployment budgets, this paper establishes a clear decision rule. If you possess a strong pretrained parent model and target retraining data is strictly limited, pruning offers a highly efficient shortcut over random initialization. However, if target training data is abundant and deployment requires dense architecture execution, pretraining a large parent solely to perform structured pruning is compute-inefficient. In that regime, spending the full token budget to train a target small dense model directly from scratch delivers equivalent or superior downstream capability.
Details
The Pretraining Shortcut Dilemma: Compression vs. Token Efficiency
The prevalent strategy for constructing performant small language models relies on compressing multi-billion-parameter foundation models through structured or unstructured weight removal. This approach rests on the assumption that a small model initialized from a trained parent inherits knowledge that cannot be easily acquired through standard pretraining at smaller scales. In the vision domain, classic investigations into convolutional networks, such as the landmark work by Liu et al. (2019), demonstrated that structured pruning often functions primarily as neural architecture search, where training the pruned target architecture from scratch achieves performance parity with the pruned-and-fine-tuned counterpart. Whether this principle extends to modern autoregressive language models trained on trillions of tokens across distinct architectural granularities has remained an open question.
To resolve this ambiguity, the authors establish a rigorous token-matched comparative framework. Previous pruning evaluations typically compared the compressed model only against its unpruned parent or a fine-tuned baseline, obscuring whether the observed performance stemmed from inherited weight values or simple architecture selection. This study isolates these effects by comparing pruned initialization against random initialization across two controlled budget regimes. The first regime enforces an equal training token budget, comparing a target model retrained for N tokens after pruning (P200−RN) against an identical architecture trained from scratch on the exact same N tokens (SN). The second regime enforces an equal total token budget, granting the scratch-trained baseline the entire token allocation consumed by both pretraining and retraining (S200+N), testing whether additional training tokens alone can close the capability gap.
