Field Note · World Models

The four learning paradigms behind a world model

23 June 2026 · Risto Anton Paarni · Helsinki, Finland

Companion to the three-piece loop. A world model for control rooms is not one model. It is four learning paradigms, each doing a different job.

Generative models build the simulator. Reward learning refines it. Reinforcement learning deploys an agent into it. Self-supervised learning compresses the raw signal so the other three can work.

1 · Reinforcement learning — decision-making

An agent observes a state, takes an action, gets a reward and a new state, and updates its policy to maximize long-term reward. In a world model the agent treats the simulator as a sandbox: it runs many imagined trials to master a control task before touching the real system.

2 · Reward learning — defining "good"

Hand-writing a perfect reward equation for the physical world is hard. So the system builds its reward from operator behaviour: an expert demonstration, or a comparison ("plan A is safer than plan B"). In our reference loop the signal is the simplest honest one — an operator override. Override an action and its type is weighted down, in the open:

// reward-store.ts — an override is a negative reward
if (feedback.verdict === "override") {
  after = Math.max(WEIGHT_FLOOR, before * (1 - this.learningRate));
} else {
  // accept nudges the weight back toward 1
  after = before + (DEFAULT_WEIGHT - before) * this.learningRate;
}

Final ranking is simScore × weight, so the loop measurably shifts toward what operators accept and away from what they reject — across cycles you can read in a log.

3 · Supervised & generative — the world engine

Generative models (diffusion, autoregressive transformers) learn the rules of the environment from paired data: given a history of frames (X) and an action (A), predict the next state (Y). That trained engine is the simulator the reinforcement agent practises in.

4 · Self-supervised — compression

Before any decision, raw inputs — pixels, dense sensor graphs — are compressed into a latent representation. Self-supervised vision and graph encoders learn that structure with no human labels, by predicting a part of the data they hid from themselves.

The honest part

The reference loop ships paradigm 2 — reward learning from overrides — as real, inspectable code. The generative simulator (paradigm 3) is a deterministic stand-in, not a trained physics model, and we say so. Calling a transparent preference weight "learning" is honest at exactly this level and no higher. That restraint is the point.

How they intersect

Generative / supervised models build the simulator.
Reward learning refines it, keeping transitions inside human and physical limits.
Reinforcement learning deploys an agent to find the safest, most efficient policy.
Self-supervised learning feeds all three with a compact view of reality.

The short version

Four paradigms, one loop: build, refine, deploy, compress.
The reward signal that matters in a control room is the operator's override.
Our reference makes that signal real code; the rest is named, not faked.
Perception → imagination → execution, closed by reward.