The proposed alternative is JEPA, the joint-embedding predictive architecture. It has three parts: an encoder, a predictor and a planner. The encoder turns each frame of observation into a learned embedding, a compact vector that describes the scene. The predictor takes that embedding plus an action and predicts the embedding of the next frame. The planner then tries hundreds of action sequences inside the predictor and picks the one whose predicted end state lands closest to the embedding of the goal. The whole loop runs in embedding space, so the model only has to learn the parts of the scene that are important for deciding actions.

This year a team including LeCun released LeWM [2], a JEPA world model whose main benchmark is Push-T [3], a simulation where a robot pushes a gray T-shaped block across a table until the T sits in the green target position shown in a goal image. I've spent the last few months running 600 experiments on it.