Gradient Dissent · Thu 22 Oct 2026 · NVIDIA's looped models The loop gate sets a step size for every token and every channel In a looped model the same middle block runs again and again. Each pass proposes a new state, and the gate decides how far to move toward it. It decides separately for every token and every channel. The figures respond to taps and drags. 0.35 every step size when training starts \(\sigma(0)\cdot\operatorname{softplus}(0) = \tfrac{1}{2}\ln 2\) ≈0.2% of the model's parameters \(d^2 + 2d\): 9.4M at \(d = 3{,}072\) −0.025 one-loop loss from the gate alone 1.660 against 1.685 on a 0.5B model 1 The update2 Where it sits3 One step4 The gate matrix5 Across loops6 At one loop7 Which rule wins8 Dory 1 · Each loop moves every channel part-way toward a proposal \[ s_i = s_{i-1} + \beta_i \odot \left(r_i - s_{i-1}\right) \] \[ \beta_i = \sigma\!\left(W_\beta\, \tilde s_i + b_\beta\right) \odot \operatorname{softplus}(\gamma) \] \(s_{i-1}\): the state before loop \(i\), a vector of \(d\) numbers for each token. It starts as \(h_{\text{in}}\), the input block's output. \(\tilde s_i = s_{i-1} + h_{\text{in}}\): what the loop block reads. The input is added back on every loop. \(r_i = \mathcal{R}_\theta(\tilde s_i)\): the loop block's proposal for the new state. \(\beta_i\): the gate, one step size for each token and channel. \(\odot\) multiplies channel by channel. \(W_\beta\) (\(d \times d\)), \(b_\beta\) and \(\gamma\): the gate's only parameters, shared by every loop. Read the gate from right to left. \(\operatorname{softplus}(\gamma)\) is a learned cap for each channel. The sigmoid, between 0 and 1, says how much of that cap this token uses on this loop, and the gate matrix \(W_\beta\) computes it from the whole state. 2 · The same block and the same gate run k times Loops 1234 Koi-4B: 14 input layers, 24 in the loop, 14 output layers. At one loop every layer runs exactly once, so the model costs the same as an ordinary 52-layer model. Each extra loop adds 24 layers of compute and no parameters. The arc over each block is the old state, which the gate blends with the block's proposal. 3 · One step: β = 0 keeps the state, β = 1 replaces it Drag the blue and orange points. β for channel 1 0.35 β for channel 2 0.35 Same β for both channels 0 keeps, 1 replaces, anything between blends. Plain looping is \(\beta = 1\): the proposal overwrites the state. Each channel steps on its own. Untick the box and give the channels different βs: the new state leaves the straight line from \(s\) to \(r\). Above 1 overshoots. That needs a cap \(\operatorname{softplus}(\gamma)\) above 1. The formula allows it, but the paper doesn't report the trained caps. 4 · The gate matrix reads the whole state to set each channel's step Row \(j\) of \(W_\beta\) is a detector for channel \(j\). It takes a weighted sum over all \(d\) channels of \(\tilde s\): a large positive sum opens the gate, a large negative one closes it. The toy below has \(d = 8\); Koi-4B has \(d = 3{,}072\). At initializationTrained (toy) TokenABC \(\tilde s\) across the top, \(W_\beta\) below, one row per channel This token's step \(\beta_j\), inside its cap Channel 12345678 Change channel 3 of the state 0 Reset this token Toy numbers with \(d = 8\), made up to show the arithmetic. They are not the paper's weights. At initialization every step is 0.35. \(W_\beta = 0\) and \(b_\beta = 0\) put every sigmoid at 0.5, and \(\gamma = 0\) puts every cap at \(\ln 2 \approx 0.69\). Training starts from one uniform step and learns where to differ. Every row reads every channel. Move one channel of the state and several gates move. So the step depends on what the token is, not only on which channel it is. The cap belongs to the channel, the sigmoid to the token. A channel with a small cap never moves far, whatever the token says. 5 · Across loops the same matrix gives a different step \(W_\beta\), \(b_\beta\) and \(\gamma\) are shared by every loop. The step still changes from loop to loop, because \(\tilde s_i\) changes as the state moves. Loops 2346 \(\beta\) for each loop (rows) and channel (columns) How far each loop moved the state Grey: the change the block proposed. Green: how far the gate let the state move. A toy loop block (a fixed random layer with tanh), with section 4's token and gate. As the state settles, each loop proposes a smaller change. Dory1.0-4B, which has no gate, settles the same way: over 16 loops its state moved 2.6% at loop 2 and about 0.3% per loop by loop 10. 6 · At one loop the gate is a learned skip around the middle block \[ s_1 = (1-\beta_1)\odot h_{\text{in}} + \beta_1 \odot \mathcal{R}_\theta(h_{\text{in}}) \] This is one loop as the paper's 0.5B ablations run it, without adding the input back. Each channel keeps part of its input and takes part of the block's output. An ordinary model has no such choice: it always takes all of the block's output. For section 4's token: how much of each channel comes from the block Orange: the share taken from the block, the number at right. Grey: the share kept from the input. The gate alone took 0.025 off the one-loop loss GPT-0.5B trained on 300B tokens, validation loss; lower is better. Ladhak et al., section 6.2. The model trained only at one loop runs every layer once, like the baseline. Without the input added back the two differ only in the gate, and the gate model is 0.025 lower. 7 · Only a learned, token-dependent step matches the baseline at one loop Rule Update Result, GPT-0.5B Replace \(s_i = r_i\) Highest one-loop loss; worse at 5 loops than at 4 Residual \(s_i = s_{i-1} + r_i\) Stops improving after 3 loops Delta \(s_{i-1} + \alpha \odot (r_i - s_{i-1})\) Second at every depth, 0.006–0.007 behind Adaptive delta, the gate \(s_{i-1} + \beta_i \odot (r_i - s_{i-1})\) Lowest loss at every loop count Delta's \(\alpha\) is learned for each channel but is the same for every token; it starts at 0.05. Ladhak et al., section 6.1. With the gate, one loop matches the baseline and five loops are 0.058 lower GPT-0.5B trained with mixed loop counts (up to 5), validation loss. Ladhak et al., Table 14. 8 · Dory, NVIDIA's released 4B looped model, has no gate matrix Koi-4B, the paper's model Dory1.0-4B, released weights Between loops A gated step, \(s + \beta \odot (r - s)\) Replace, then normalize: \(\operatorname{norm}(\mathcal{R}(\operatorname{norm}(s + h_{\text{in}})))\) Inside each layer Ordinary residual layers nGPT's step, \(h \leftarrow \operatorname{norm}(h + |\alpha| \odot (f(h) - h))\) Who sets the step Each token and channel, through \(W_\beta\) Each channel, through \(\alpha\); the same for every token Gate parameters \(W_\beta\) is \(3{,}072 \times 3{,}072\), plus \(b_\beta\) and \(\gamma\) None among its 507 tensors Koi's step changes with the token; Dory's does not Koi-style: \(\beta\) from \(W_\beta \tilde s\) Dory-style: \(\alpha\) for each channel Toy values; rows are tokens, columns are channels. Each panel has its own colour scale. Dory's per-layer step is the paper's "delta" rule, moved inside every layer. In the paper's 0.5B ablation, delta came second, 0.006–0.007 behind the gate. The question for the room: does nGPT's per-layer step already do most of the gate's job? Not the same gateKoi's Gated DeltaProduct layers have gates of their own, which decay each layer's running memory from one token to the next. The loop gate acts between loops, on the whole residual state. More for Thursday: the 17 slides, the 10-minute audio overview and the companion write-up. Sources and notes Faisal Ladhak, Shantanu Acharya, Santiago Akle Serano, Simeng Sun, Boris Ginsburg, Krishna C Puvvada, "One Model to Loop Them All: Elastic Latent Recurrence for Language Models", arXiv 2610.12582, 8 Oct 2026. The gate and its initialization: Eq. 2 and section 3.1. Update rules: section 6.1 and Figure 4a. The one-loop result: section 6.2 and Figure 4b. Losses with the gate: Table 14. Parameter share and hidden sizes: Appendix A and Table 4. Dory1.0-4B: the tensor list is from its safetensors header at revision b366b87f, read Tue 20 Oct 2026. The loop's form agrees with NVIDIA's vLLM implementation. The 16-loop settling figures come from an independent re-measurement of the released model. nGPT: Ilya Loshchilov, Cheng-Ping Hsieh, Simeng Sun, Boris Ginsburg, "nGPT: Normalized Transformer with Representation Learning on the Hypersphere", 2024. The interactive figures use toy numbers with \(d = 8\). They show how the arithmetic works, not a trained model's values.
