Weight space editing changes a neural network’s knowledge, behaviour and alignment through direct modification of its weights. Model-mediated weight space manipulation, also known as weight space programming or intra-weight injection, has emerged as a capability of advanced frontier models. While it offers benefits for gradient-free adaptation, it also creates an opportunity for one model to induce unauthorised behaviour in another. We call this weight space misfolding, by analogy with protein misfolding in prion disease, and consider both direct interference and hypothetical onward propagation. The asymmetry between an attacker seeking a harmful intervention, who merely needs to generate the payload attack once, and a defender who needs to find that needle in the haystack of weights, creates an impossible choice for the latter: they can either prohibit weight editing altogether, forfeiting its benefits, or be bound to cover the burden and cost. We develop a threat model and a non-exhaustive taxonomy covering factual substitution, capability denial and selective omission, including offensive uses of selective unlearning, then illustrate it on the basis of three selected case studies using the IWIBench interface. We find that while targeted disruption is possible,[p7] the limitations in reliability and functional preservation are significant.[p8] While the cases explored in this paper do not demonstrate self-propagation or actual offensive capabilities, it is abundantly clear that the building blocks are present and operate reliably in frontier models. While these cases by no means directly establish self-propagation, they clearly show the evidence These cases do not demonstrate self-propagation, but show why imperfect editing capabilities can still pose a security risk. We examine defensive approaches, including control of the path from authorised editing to deployment, acceptance and recovery requirements and the implications for physical AI.
Models can now transmit viruses in weight-space

Key takeaways
Models can now transmit viruses in weight-space, posing security risks through unauthorized behavior changes.
- Weight space editing allows models to alter knowledge and behavior directly.
- This creates risks as attackers can induce harmful behavior in other models.
- Defensive measures face challenges due to the asymmetry between attackers and defenders.
Summarised automatically by AI from the original article by Hacker News. AI can make mistakes, so check the original for details.
Story details
- Published
- By
- mkglees
- Format
- Article
- Original
- alphaxiv.org ↗

