Skip to slides GRADIENT DISSENT / THU 22 OCT 2026Paper ↗Dory card ↗Dory re-measured ↗Contents NVIDIA · arXiv 2610.12582 · posted Thu 8 Oct 2026 · 26 pages One model, many depths. One Model to Loop Them All: Elastic Latent Recurrence for Language Models. A looped language model that costs nothing extra at one loop and gets better with every loop you add. Then Dory1.0-4B, the model NVIDIA says was built on this recipe, re-measured. Faisal Ladhak · Shantanu Acharyaequal contribution Santiago Akle Serano · Simeng Sun · Boris Ginsburg · Krishna C PuvvadaNVIDIA · Sun and Ginsburg also wrote nGPT (2024) with Ilya Loshchilov → / space / swipe to advance · o contents · f fullscreen. Paper numbers come from its tables; Dory numbers from NVIDIA's model card and your two Modal experiments (19–20 Oct). Hover or tap a mark for its value. The idea A loop adds depth without adding parameters. Loops 2 The input block's output hin is added back in at every loop. At one loop every layer runs exactly once. Effective depth D(k) = Lin + k·Lrec + Lout. Koi-4B splits its 52 layers 14 / 24 / 14. Dory splits 72 layers 16 / 40 / 16, which is 8 / 20 / 8 attention-plus-MLP blocks. Memory is per-loop KV for attention layers inside the loop; with last-loop caching it stays at 1×. The paper's word is elastic depth: one checkpoint, served at any loop count it was trained on. It works like a reasoning-effort setting, except the extra thinking happens in the hidden state rather than in more tokens. The problem Looped models have paid a one-loop tax. 34.9 vs 46.4Huginn (mean recurrence 32) against its fixed-depth baseline on ARC-Easy, after 800B tokens below → aboveParcae: under its Transformer baseline at 1 loop, over it at 8 +0.0225nats above the baseline at 1 loop, sampling 1–5 loops uniformly (GPT-0.5B ablation) If one loop is worse than a plain model of the same size, the latency tier goes to a separately trained plain model. The goal here is narrower than "is looping compute-efficient?": make the model elastic without the tax. A gated state update with a learned step size per token and channel. Loop counts mixed inside each optimizer step: a rotating slice of data-parallel ranks always runs one loop. Each pass told where it is: conditioning on loop index and budget. Plus three ways to keep it cheap: switch to mixed depths late in pretraining, reuse the first loop's MoE routes, keep only the last loop's KV cache. Every ELR model trains with its baseline's optimizer settings unchanged. Ingredient 1 · the state update Take a gated step toward the block's proposal. Don't replace the state. s̃i = si−1 + hin · ri = Rθ(s̃i) si = si−1 + βi ⊙ (ri − si−1) βi = σ(Wβ s̃i + bβ) ⊙ softplus(γ) Rule (GPT-0.5B) What happens Replace: si = ri Highest one-loop loss; gets worse from 4 to 5 loops Residual: si−1 + ri Stops improving after 3 loops Delta: fixed per-channel α, init 0.05 Second at every depth, 0.006–0.007 behind Adaptive delta (β above) Only rule matching the baseline at 1 loop; lowest loss at every depth Wβ, bβ start at zero and γ at 0, so every step starts at σ(0)·ln 2 ≈ 0.35. Cost: d² + 2d parameters, about 0.2% of a 4B model. Step β 0.35 Schematic. β = 1 is "replace". Dory's nGPT backbone takes a step of this shape inside every layer, h ← norm(h + α(f(h) − h)) with a learned per-channel α (initialised at 0.05), but has no gate at the loop boundary. Ingredient 2 · loop sampling Mix loop counts inside every update, not across updates. GPT-0.5B, 300B tokens, validation loss at one loop (lower is better); the dashed line is the non-recurrent baseline, 1.6852 (Table 3). "Independent" lets each rank draw its own depth, so every step waits for the deepest rank: 1.36× step time. Huginn draws one loop count per step for all ranks. Even with one loop drawn a third of the time, the one-loop model stays above baseline (1.6871). Forcing a rotating fraction f of ranks to run one loop puts one-loop and deep gradients in the same update, at no step-time cost. Override fraction f 10% Max loops K 45 p1 = 1/K + (1 − 1/K)·f and k̄ = f + (1 − f)(K + 1)/2 (Eq. 3). The paper picks f so that p1 ≈ ⅓: 15% at K = 5, 10% at K = 4. Fixed depth vs mixed depth A model trained at one depth breaks at every other depth. Validation loss, GPT-0.5B (Figure 4b). Rows: the depth a model was trained at. Columns: the depth it is run at. Outlined: train = run. Non-recurrent baseline 1.685. Specialists win at home (1.603 at 5 loops, against 1.627 mixed) and collapse away from it: the 1-loop model reads 4.242 at 5 loops. The mixed model improves monotonically, 1.683 → 1.627, and serves five depths with one checkpoint. A surprise: the model trained only at one loop beats the plain baseline, 1.660 against 1.685, with the same layers run once. At one loop the gate is a learned skip around the middle block, and "the gate appears to help even without recurrence." The paper's open question: conditioning closes most of the gap to the specialists at intermediate depths but not at the endpoints. Ingredient 3 · conditioning Tell each pass which pass it is, and how many are coming. ci = MLP[ φ(i / k) ; φ(k / K) ] x̃ = (1 + λ(ci)) ⊙ RMSNorm(x) + μ(ci) y = x + (1 + ω(ci)) ⊙ f(x̃ + A(g(ci) ⊙ Bx̃)) AdaLN/FiLM-style scale and shift, a low-rank correction with a loop-dependent gate, and an output rescale, on every recurrent and output layer. Progress τ = i/k and budget κ = k/K follow LoopFormer. Grafted at the switch, starting from zero. Every head starts at zero, so the conditioned model is exactly the one-loop checkpoint at every k, and the deeper passes learn to depart from it. −0.022loss at 5 loops (−0.009 at 2); one loop moves 1.6748 → 1.6739 0.038 → 0.059total loss drop from 1 to 5 loops, without → with conditioning 1.6147at 5 loops, delayed + conditioned, against 1.6273 for mixed depth from scratch +3.9%parameters in Koi-4B (155.6M); +6.3% at GPT-0.5B (31.6M) A trade-off the paper notes but doesn't test: with a budget input, loop i depends on k, so a request can't stop early mid-computation. They serve each request at a fixed k and leave adaptive depth to future work. Keeping training cheap Train at one loop for most of the run. Switch to mixed depths late. Lines: (1 − ρ) + ρ · D(k̄)/D(1) (Eq. 5) for the two layouts. Dots: GPT-0.5B's measured runs, labelled with the share of the five-loop improvement each keeps (Figure 5a). Share of tokens at mixed depth ρ 25% 4B and 30B-A3B: last 25% mixed → 1.16× the baseline's training compute, against 1.62× mixed throughout. Koi-4B: last 50% of 7T tokens mixed → 1.31×. GPT-0.5B: 1.79× from scratch; a 75% start costs 1.20× and keeps 84% of the five-loop gain, for +0.009 loss at five loops. One loop stays below baseline at every start point. Results · same data, same layers No tax at one loop; four loops add about five to seven points. Ten-task average (MMLU, MMLU-Pro, ARC-C, GSM8K, WinoGrande, CSQA, RACE-high, HumanEval, MBPP, MATH-500). Dashed: the non-recurrent twin. Tables 8–9. −0.52 / +1.04one loop against the twin: 4B dense / 30B-A3B MoE +7.25 / +4.75four loops against one loop, same checkpoint 79% · 96%of the 1→4 gain reached at 2 and 3 loops (4B) 1T / 300Btokens: the 4B and the 30B-A3B pairs Both are GDP-Hybrids: 23 Gated DeltaProduct + 6 attention + 23 feed-forward layers, split 14 / 24 / 14. At one loop single tasks swing both ways (4B: GSM8K −5.3, MATH-500 +5.2), and every number is a single run, so a half-point gap is hard to call. Largest 1→4 gains: GSM8K +11.8, MATH-500 +11.3 (4B). What a loop costs to serve Loops cost latency; last-loop KV keeps memory flat. 2.06×decode latency per token at 4 loops (4B, H100, batch 1); 1.36× at 2 loops 2.33×time to first token at 4 loops; 3 loops reach 96% of the gain at 1.87× −22% → −33%Koi's output tokens at 2 → 4 loops, so a rollout's decode time is +6% at 2 loops, ×1.4 at 4 Last-loop KV cache One cache per attention layer, overwritten each loop; a loop-i query reads earlier tokens' final-loop keys. Koi at 4 loops: 3.0 GiB instead of 7.5 GiB per 128K-token request (BF16), for 0 to −3 points (AA-LCR −3.0, HLE −1.4, SciCode −1.4). No retraining. Cross-loop route replay (MoE) Route each token once, on loop 1, and reuse the experts later: router calls 53 → 23 per 4-loop pass, update time −11.7% (−19.6% at fixed k = 4), loss +0.0038. Inference gains only 1.2%. Tokens keep just 53% of their loop-1 experts by loop 4, yet replayed outputs keep cosine 0.77 with fresh ones. The full recipe, post-trained Koi-4B: 18 at one loop, 29 at four. AA Intelligence Index v4.1 estimates from NVIDIA's own harness, not Artificial Analysis (Table 10, Figure 3). CritPt is scored zero. The second loop is 8 of the 11 points. Two loops sit above Qwen3.5-9B and level with Gemma 4 26B-A4B (6.5× the parameters). Training: 7T tokens at 8K, one loop for the first half; 50B tokens of 256K context extension; 4 epochs of SFT; 200 RLVR steps at a 4 : 1 : 1 : 1 ratio of 1–4 loops, with a pass-rate curriculum that sends harder prompts to more loops. Per benchmark, 1 → 4 loops: GPQA 59.9 → 72.8, IFBench 61.2 → 71.8, Terminal-Bench 12.1 → 28.0, SciCode 17.7 → 33.4, AA-LCR 51.3 → 68.3. Koi has no matched baseline. Its one-loop score is compared with other open models, so data and post-training differences are folded in. The released model Koi is not Dory. Dory's loop has no gate and no conditioning. Koi-4B · the paper Dory1.0-4B · released Fri 16 Oct Backbone GDP-Hybrid: 23 Gated DeltaProduct + 6 attention + 23 MLP layers, no position encoding nGPT, the normalized transformer: 36 attention + MLP blocks, 27 with a 1,024-token sliding window, 9 with full attention, RoPE Input / loop / output 14 / 24 / 14 layers 16 / 40 / 16 layers (8 / 20 / 8 blocks); 5 of the 9 full-attention layers sit in the loop Loop update Adaptive-delta gate No gate: s ← norm(core(norm(s + hin))). nGPT's per-layer α steps act inside Loop conditioning Yes, +155.6M parameters None. The 507 tensors hold only per-layer α vectors; loop i is identical at every k Pretraining 7T tokens, BF16 6T tokens, NVFP4 for the first ~4.5T, then BF16; 300B tokens of context extension Default — 2 loops; 1–4 supported GPQA · 1 / 2 / 4 loops 59.9 / 69.9 / 72.8 69.19 / 72.41 / 72.16 IFBench 61.2 / 70.1 / 71.8 74.29 / 77.94 / 79.0 AA-LCR 51.3 / 65.7 / 68.3 63 / 71 / 73 Terminal-Bench 2.1 12.1 / 22.9 / 28.0 12.5 / 18.1 / 22.2 Koi: the paper's Table 10 (its harness). Dory: the model card (NeMo Gym), config.json, and the safetensors header at revision b366b87f, read Tue 20 Oct. Your independent PyTorch forward pass, built from NVIDIA's vLLM PR and llama.cpp fork, matches vLLM to 0.0002 nats with no gate and no conditioning. NVIDIA (Shantanu Acharya, 20 Oct): Dory was "built on" this paper's methodology and training recipe; the Dory tech report is due by the end of the month. Dory is stronger at one loop and gains less from looping. Your experiments 1 and 2 · Mon 19 – Tue 20 Oct · Modal Dory reproduces its card, and only the second loop pays. Blue: your runs, single pass at a 64K output budget, bars = 95% interval. Orange: NVIDIA's card. GPQA Diamond (4–6 repeats), IFBench on NVIDIA's metric (mean strict fraction, 2–7 repeats), AA-LCR (2–3 independent draws of 100 questions). +5 · +4 · +9points from the second loop: GPQA, IFBench, AA-LCR, for 1.39× the compute per answer ≈ 0from loops 3–4: 2→4 is −1.0 ± 1.0, +0.2 ± 0.8 and +0.7 ± 2.7 1 pass ≈ 5–6 votesone 2-loop GPQA answer (73.2%) matches a majority vote of five or six 1-loop samples, at about ¼ the compute Inside Dory · one 16-loop pass, read out after every loop Loop 1 rewrites the state; loop 2 polishes; then it drifts to a worse fixed point. Teacher-forced loss, 70,192 tokens from 24 documents, by the loop the output is read after (your independent implementation). How far each loop moves the state (relative L2), log scale. Loop 1 moves it 142% (cosine with the input −0.02), so it is left off. Each loop count is its own distribution. Text Dory writes at k loops is predicted best at k loops; the 1 → 2 shift is about
