We changed requirements in a coding benchmark and found models talking themselves out of following them.
Sometimes the agent understood the new requirement perfectly. It described what needed to change, considered an implementation, and then decided the hidden tests probably wanted something else. In one run, a model implemented the change correctly, asked another model for the benchmark’s original answer, and removed its own correct code after receiving it.
This is an awkward habit for a coding assistant to have. It is also a habit that a familiar benchmark can reward. If the model has guessed what the tests expect, it gets the same passing result as an agent that worked carefully from the requirements.
