Loop the Loopies!
Authors: Zitian Gao, Yilong Chen, Yihao Xiao, Xinyu Yang, Ran Tao, Joey Zhou, Bryan Dai
Paper: https://arxiv.org/abs/2607.16051
Code: https://github.com/IQuestLab/loopie/megatron, https://github.com/IQuestLab/loopie/vllm (404 for now)
Model: https://huggingface.co/IQuestLab/Loopie-20B-A2B-preview, https://huggingface.co/IQuestLab/Loopie-6B-A0.6B-preview (404 for now)
TL;DR
WHAT was done? The authors from IQuest Research introduce the Loopie series, a family of looped Mixture-of-Experts (MoE) models that replaces standard deep architectures with compute-matched recurrent depth. By employing a localized “layer-loop” topology and a novel Supervised Pre-Training (SPT) paradigm, they trained models up to 20B parameters (2B active) that rival significantly larger networks.
WHY it matters? Looped transformers have historically been evaluated based on parameter efficiency, ignoring the fact that recurrent passes multiply pre-training compute. This paper demonstrates that when strictly controlling for total training FLOPs—trading stored depth for microbatch efficiency—recurrent depth acts as a highly competitive, hardware-friendly scaling axis that achieves frontier-level reasoning capabilities with a fraction of the data.
Executive summary: For research leaders projecting scaling laws, this paper challenges the default strategy of non-recurrent depth scaling. The “Loopie Recipe” proves that halving physical depth, doubling microbatch sizes, and looping layers twice (R=2) yields superior returns on compute. Paired with a massive-scale post-training regime that mitigates catastrophic forgetting, Loopie models reach state-of-the-art math and code performance, suggesting that explicit internal recurrence should be a primary architectural consideration for next-generation reasoning systems.
Details
The Compute-Accounting Bottleneck
The field has long recognized the theoretical elegance of recurrent architectures, from Universal Transformers [review] to recent models like Huginn and Ouro [review]. These designs reuse parameters across depth, creating a powerful inductive bias for algorithmic reasoning and iterative refinement. However, the adoption of looped models for frontier large language models (LLMs) has been bottlenecked by a fundamental compute-accounting flaw. A model looped N times during training requires N times the pre-training compute. Consequently, comparing a looped model to a vanilla model with the identical parameter count creates a false equivalency. The true test of recurrent depth is whether it can outperform non-recurrent depth when given the exact same pre-training compute budget and wall-clock time. The authors address this by explicitly shifting the evaluation paradigm from parameter-efficiency to compute-efficiency, comparing their proposed MoE models against vanilla baseline networks configured to consume the same computational resources.
