Authors: Ziyue Li, Yang Li, Tianyi Zhou
Affiliations: University of Maryland, College Park; Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)
Paper: https://arxiv.org/abs/2606.06574
Code: https://github.com/tianyi-lab/PoLar
Model: N/A
TL;DR
WHAT was done? The paper introduces Program-of-Layers (PoLar), a training-free framework for pretrained Large Language Models (LLMs) that treats transformer layers as a discrete library of callable functions. Instead of routing every input through a rigid, sequential forward pass, PoLar predicts an input-specific program that dynamically skips, keeps, or repeats contiguous layer segments using a lightweight, 2.1-million-parameter predictor network.
WHY it matters? This work demonstrates that fixed-depth, static-order inference exposes only a fraction of an LLM’s intrinsic reasoning capability. By dynamically compiling layer-level execution paths at test time, PoLar proves that latent reasoning can be systematically scaled and reallocated, achieving notable accuracy gains and latency reductions without modifying or fine-tuning any base model weights.
Executive summary: Modern AI models run the exact same computational steps for simple questions as they do for extraordinarily difficult mathematical problems. PoLar provides a meta-controller that plans how to execute an existing, frozen AI model for each specific question—skipping unnecessary internal steps on easy queries to save time, or repeating specific analytical steps on hard queries to improve accuracy. This architectural steering reduces operational latency while substantially improving performance across complex reasoning benchmarks, all without requiring expensive model retraining.
Details
The Static Forward-Pass Bottleneck
Modern autoregressive transformer architectures process every input token sequence through an invariant sequence of D feed-forward and self-attention operations. While natural language queries exhibit high variance in algorithmic complexity and requisite reasoning depth, the canonical inference pipeline uniformly enforces an invariant computational budget. When confronted with simple lookups or syntactically repetitive tasks, a static forward pass over-computes; conversely, when confronted with complex multi-step reasoning, the fixed layer depth frequently proves insufficient unless computation is offloaded into extensive token-level chain-of-thought generation.
Existing dynamic computation paradigms attempt to mitigate this inflexibility but operate under restrictive structural paradigms. Static pruning techniques such as ShortGPT permanently excise layers to reduce throughput latency at the expense of peak model capacity, while input-adaptive routers like MindSkip and FlexiDepth introduce token- or sequence-level early exiting that only implements monotonic truncation. Emerging dynamic routing frameworks, including DR.LLM, introduce layer repetition but interleave local routing decisions during execution, limiting global depth coordination. PoLar resolves this structural disconnect by demonstrating that layer skipping and layer recurrence are fundamentally complementary operations that must be orchestrated globally at the program level rather than through isolated, local decisions.
