- A researcher at elonlit.com showed that a loss term logging exactly zero can still make PyTorch's backward pass rewrite chosen model weights during training.
- Hash checks, code skims and flat training curves would all pass, so a training run has to be treated as code execution. This is a proof of concept, not a reported attack.
- Ask who reviewed each custom loss function and compiled extension, and whether weights that should not change are compared before and after training.
A zero on the dashboard is not proof that nothing happened
A typical sanity checklist for fine-tuning an open model looks like this. Check the file hash. Skim the model code. Watch the training curve and confirm that extra logged values stay flat. The researcher's own scenario begins with exactly these steps.
A researcher publishing at elonlit.com has shown that all three checks can pass while a hidden program runs. The lesson here is simple. A training run is code execution, and a flat line in a log describes the forward pass only. It says nothing about what the backward pass does.
What the researcher demonstrated
The researcher built a loss term that evaluates to exactly 0.0 in every metric, yet forces gradient descent to overwrite chosen weights. The weights hold the state of Conway's Game of Life, and each optimizer step advances the board by one generation.
The researcher then tried it inside a working model: a convolutional network of about 1.6 million parameters, trained on the MNIST dataset. The zero-valued term controlled 1,024 weights in one hidden layer. Across 5 epochs and 2,345 training steps, the extra loss read 0.0 every time.
The classifier reached 98.80% test accuracy, against 98.84% for an untouched baseline. According to the researcher, the 1,024 weights played all 2,345 generations of the game without a single cell error.
This is a proof of concept written by the researcher. The post does not report any attack of this kind in the wild.
How the trick works
Computers store decimals in fixed-size formats. In the common 32-bit format, numbers near 2^30 (about 1.07 billion) sit 128 apart. Add a small number such as 1 to 2^30 and the hardware rounds it away. Subtract 2^30 afterwards and the result is 0.0.
PyTorch's automatic differentiation, the machinery behind loss.backward(), does not see that rounding. It records an addition and a subtraction and applies a textbook rule: each has a slope of 1. So the forward pass sees zero while the backward pass sees the original number.
The researcher combines two such pieces. One blocks gradients without calling the visible .detach() function. The other multiplies the update by a zeroed weight. The forward result is zero. The backward pass delivers a gradient equal to the current weights minus the desired next state. With plain SGD at a learning rate of 1, the optimizer then subtracts that gradient, and the weights become the next state. The researcher says other learning rates need the gradient rescaled.
Why a code review would miss it
The researcher describes three ways to hide the logic. The first is to dress the update rule as a geometric regularizer, a standard stabilizing term, so it reads as ordinary math. A 3×3 convolution followed by a threshold is the same shape as a normal deep learning layer.
The second is to store the constants in a config file described as the output of a hyperparameter sweep. The third is to ship a compiled CUDA extension. The researcher notes that engineers routinely install precompiled binaries with libraries such as flash-attn and bitsandbytes. In that case, the logic sits in compiled GPU code and the Python repository shows only a thin wrapper.
The base checkpoint stays untouched. Its hash still matches. The researcher argues this also sidesteps static scanners, which inspect weight files before training starts, because the payload is only assembled later.
What is shown, and what is only argued
Conway's Game of Life is Turing-complete, meaning it can in principle compute anything a general-purpose computer can. The researcher argues that this lets the weights serve as working memory and each training step as a tick of a processor clock. The demonstrations were Game of Life boards and a small state-machine cycle; no general computation was shown.
The researcher says the method probably works with the Adam optimizer too, using a different construction. That is stated as likely, not tested in the post.
The frontier-model scenario is also a proposal. The researcher points out that Llama 3 has 128,256 vocabulary tokens, and tens of thousands may never appear in a given fine-tuning set. Those unused rows could serve as a hidden scratchpad. The finished payload would then be written into an active part of the model.
Questions to put to your ML team
Treat third-party training code as you treat any dependency that runs in production. Ask who reviewed each custom loss function, and whether anyone has opened the compiled extensions it calls.
Ask whether the team compares weights that should not change, such as unused embedding rows, before and after training. Unexpected movement there would be a signal that a flat loss curve cannot give.
A matching hash proves you received the file you intended. It does not prove that training did only what you intended. A zero on the dashboard is a measurement of one pass, not a verdict on the whole run.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: elonlit.com.
