Frozen encoders throw information away.
So how does a pixel come back?

The recipe: a latent, a decoder, a diffusion model

"Before the chase — what exactly is RAEv2's latent, and what is this 'K' everything hangs on?"

RAEv2 freezes a DINOv3-L encoder and reads out its last K transformer blocks — each LayerNorm'd, then averaged into a single 16×16×1024 latent. K is the one recipe knob that matters here: K=1 takes only the last block (the most semantic, lowest-detail features — this is the latent RAEv2 actually generates from); K=7 takes every other block from 11 to 23 (its reconstruction default); K=23 takes all of them (the most detail). Deeper blocks hold more meaning and less surface detail, so K sets how much low-level detail the latent carries — the same 16×16 grid throughout; only which layers are summed changes.

image

Σ+ LN

latent · K=1

Each tile is one DINO block's features (PCA-coloured), piled into a 24-block deck. Press a K — the summed blocks lift out of the deck and merge into that K's latent on the right. K=1 is the last block alone (sparse, semantic); K=23 folds in every block (rich, detailed).

On that latent, two training stages. Stage 1 trains a decoder (a ViT-XL) to reconstruct pixels from the frozen latent. Stage 2 trains a diffusion transformer (DiT) to sample new latents in the same space; the very same decoder then renders those samples into images. So the latent does double duty — it must be faithfully reconstructable (Stage 1) and cleanly generatable (Stage 2) — and the tension between those two jobs is what the rest of this analysis pulls on.

pure
noise → → a valid latent in
the encoder's space

Stage 2 trains a diffusion model (DiT) to do exactly this — start from noise and arrive at a latent that belongs to DINO's space, which the decoder then renders into an image. (Illustrative loop.)

With the recipe in hand — start by asking what each encoder layer still remembers of the pixels.

There is an order to what gets discarded

"If the encoder throws information away, does it vanish all at once — or layer by layer?"

First, a simple question: how much of the pixel does each encoder layer still remember?

Take the features of a single layer and reconstruct the original pixels from them alone (closed-form ridge). If it reconstructs well, that layer still holds the detail; if it smears, the detail is already gone.

Fit a closed-form linear (ridge) map from one layer's features to pixels, then reconstruct held-out images. Sharp means that layer still holds the detail; blurry means it is no longer linearly recoverable there. The slider sweeps L_n across all 24 layers.

original

L0 reconstruction

So the natural next question — does the decoder walk this hierarchy in reverse?

The decoder runs the encoder backwards

"If the encoder descends detail → meaning, does the decoder climb meaning → detail?"

Put this in the language of frequency, and the question sharpens. If the encoder is a low-pass filter that keeps dropping high frequency (detail), the decoder should be a high-pass filter that keeps synthesizing it.

Frequency: 2D FFT of each layer's feature map, radial-averaged into a high-frequency ratio. CKA: linear Centered Kernel Alignment between every encoder layer and every decoder layer — a mirror shows up as an anti-diagonal.

Does the same hold at the representation level? Linear CKA between encoder layers and decoder layers, drawn directly.

Two things in this CKA are worth pausing on. First, the trace never reaches L0 and stays there — every K rushes up to the detail layers, then settles back to L5 just before the readout. This "return to a canonical mid-level" is a real, still-unexplained observation. Second, the mirror is a direction, not a rigid 1:1 grid: the freely-trained RAEv2 decoder organizes itself — it races through the transition and dwells at L5 — and an even layer-by-layer correspondence only appears when one forces it through distillation. The mirror is emergent in direction, imposed in its evenness.

If the mirror creates detail — what comes back, and what never does?

What comes back, and what never does

"Texture can be plausibly invented, but text must be exact. Can the decoder bring text back?"

Varying the range of layers summed into the latent (K), I reconstructed an image containing text — because text cannot be faked plausibly, it has to be exact. Toggle K below.

The latent is the sum of the last K encoder layers (K=1: only the last; K=23: all of them). A pretrained RAEv2 decoder reconstructs from it. Toggle K to see how much of the text survives.

original

reconstruction

But that was one image. When detail doesn't come back — is it gone from the latent, or just unread?

Is the detail missing — or just unread?

"The text was one image. Push high frequency in directly: does the latent lose it, or can the decoder simply not read it?"

RAEv2 makes a striking bet — rebuild full-resolution pixels from a latent that is almost pure meaning — so something has to put the detail back. To find out what, we started with the simplest high-frequency image there is: black-and-white stripes, run through encode→decode. As the stripes get finer, they dissolve. The text case (last chapter) sharpened the puzzle but left it open: when detail fails to return, is it gone from the latent, or present but unread by the decoder? To settle it we split the reconstruction into two separable quantities — energy (is there detail at this frequency at all?) and coherence γ (is it in the right place?). Blur is low energy; sharp-but-wrong is energy without coherence — a hallucination.

One tempting explanation to rule out first: maybe the latent keeps meaningful high frequency (the aligned edges of real objects) and drops only random texture. If so, the loss would be about predictability, not frequency.

Take a natural image's detail in a frequency band, and a phase-scrambled twin with the identical magnitude spectrum — same energy at every frequency, only the phase (the structure) randomized. If meaning matters, natural should survive where scrambled dies.

From the same reconstruction, two numbers per band: E (energy) = how much amplitude returns (phase-blind), γ (coherence) = how well its phase matches the truth. This splits the returned energy into the part aligned with the true phase (γ·E, faithful) and the orthogonal remainder (E·√(1−γ²)) — energy with no correct phase, which we read as hallucination. (Throughout: held-out natural images, grayscale.)

If the gap is the latent's, reading harder shouldn't help. There are two ways to read harder without changing the latent: retrain the deployed decoder with a detail-targeted loss, or — freezing the decoder entirely — learn a per-direction amplifier on the latent (the literal "boost the high frequency" idea). Both leave the encoder frozen, so the latent codes themselves never change.

So the latent's detail is fixed by K — then why not just raise K all the way?

Pack it all in, and generation collapses

"Reconstruction keeps improving, so why does RAEv2 use only K7? Is there a reason not to pack more?"

For reconstruction alone, bigger K is better. But this latent is also meant for generation. Place reconstruction (rFID) and generation (gFID) side by side, and they move in opposite directions.

The reason lies in diffusion. Diffusion only synthesizes what is encoded in the latent. Raise K to push high-frequency detail into the latent, and diffusion must now generate that high frequency — but high frequency is near-random, with no low-dimensional structure for diffusion to model, so generation breaks down. Better to keep the latent semantic (small) and let the decoder hallucinate the detail.

Even the compromise K7 is not perfect. What does K7 miss versus K23, in frequency?

So who, inside the decoder, actually does this hallucinating?

Inside the decoder: who makes the detail?

"If detail is hallucinated, which part does it — and why does it blur?"

Split the decoder into two parts and train them separately. The head is a pure readout (an unpatchify layer); the body is the stack that climbs the hierarchy. Hand the head the real detail (the encoder's L0) and it reconstructs almost perfectly — a transparent readout. But let the body predict that detail by distillation and feed it to the head, and the result blurs.

ceiling gives the head the real detail (L0) directly — it isolates how good the readout is. composition makes the body predict L0 from the semantic latent (L23) by MSE, then reads it out — it isolates how good the body's hallucination is. Same head in both.

So — can a decoder be built to hallucinate on purpose? We tried.

Building a decoder that mirrors, on purpose

"If the decoder is the encoder's mirror, can we train one that way on purpose — and does it actually help?"

Everything so far was observation on a frozen, pretrained RAEv2 decoder. The natural step is to build a decoder that is explicitly the encoder's mirror: 23 transformer blocks, each block j supervised to match encoder layer L(22−j) by MSE — layer-paired distillation — with a transparent unpatch head and a pixel L1 + LPIPS + GAN loss.

The body is forced into the encoder's mirror by distillation; the head is a pure readout, so the pixel GAN flows through it straight into the body's hallucination.

Two questions, two clean ablations (held-out, small scale). Does distillation help? and does the mirror order matter?

Distillation slashes rFID 3× (144.6 → 47.9). But with a CNN head, shuffling the layer-pairing barely moves the number (47.9 vs 51.0) — the head absorbs the detail, so the body's mirror order stays invisible. Force a pure-readout (unpatch) head, and the order suddenly decides everything:

So we ran the same isolation at full scale — the identical 23×1024 decoder, distillation on versus off, 40k steps on full ImageNet, everything else held fixed — to see whether the small-scale gain survives. It doesn't. It reverses.

One last place to look — where does the other latent live?

Where the VAE latent lives

"Diffusion usually runs on a VAE's latent, not DINO's. Where does that latent sit on this same map?"

To pin down where low-level detail lives, it helps to bring in the latent that is built to keep it. Latent-diffusion models almost never generate from DINO — they generate from a VAE: a coder–decoder pair (here SD-VAE) made for generation, whose 32×32×4 latent is the standard carrier of low-level, pixel-near detail. So we examined it as a reference point — the explicit representative of "surface detail" — and asked where it falls on DINO's depth axis. RAE's latent sits at the semantic, low-detail end; the VAE's is the opposite philosophy, all surface. Rivals, or two ends of one axis? We measured three things — representational similarity (CKA) between SD-VAE's latent and every DINOv3 layer, linear decodability (can a single linear map turn one into the other?), and — most precisely — canonical correlation: how many of the VAE's dimensions have a linear twin inside DINO.

If the two latents are two ends of DINO's depth axis, the VAE should look like DINO's shallowest layers and nothing like its deepest. CKA tests the resemblance; the linear probe tests whether one can actually be rebuilt from the other.

Linear ridge probe, held-out, train/test split by image. Matched 16×16 resolution; absolute R² is capped by the VAE's four channels, so read the gaps, not the heights. The pixel-patch ceiling is the same probe from raw pixels — it says how much a linear map can do at all.

The same four canonical directions as 16×16 spatial maps. Pick a held-out image — top row is the VAE's, bottom is DINO L0's. They carry the same structure (sharpest for components 1–2, ρ ≥ 0.95): read through the rotation CCA found, DINO's shallowest layer is the VAE latent.

Now step back — the pieces line up.

What the experiments showed

The inverse is not a function

Beneath all of it sits one fact about inverses. The encoder is many-to-one: many images collapse onto the same meaning, their detail discarded. So its true inverse is one-to-many — a single latent ought to fan out into many plausible details, not one. A deterministic decoder can only pick a single point of that fan, which is exactly why the MSE mean blurs (it averages the fan) and why a generative loss is needed (it commits to one sample of it).

"Reconstruction," then, was never really an inverse function. It is a renderer that hallucinates a world consistent with the meaning it was handed. Generation is perception run backwards — and the backward run must invent what the forward run threw away.