MP4 MP4 Analyzer by MediaOnDeck

// engineering · methodology

Why pixel export has to be verified on YUV planes

Diffing RGBA produced a per-pixel maximum difference of 97 of 255 on 10-bit material — a number that looks like a serious bug — while the whole-frame mean for the same comparison stayed under 1.4. The contradiction was not in the code but in the measurement: RGBA already contains a layer we never meant to verify.

1. Two numbers that contradict each other

The task was to add a path exporting decoded pixels from wasm, so frame preview can fall back to a software decoder. Verification meant diffing against ffmpeg's command-line output. The first attempt compared RGBA: convert both sides to RGBA, then compare pixel by pixel.

ClipResult
8-bit sourceFine
10-bit sourcePer-pixel maximum difference 97 of 255

Read alone, 97 means "serious bug". But in the same comparison the whole-frame mean absolute difference was under 1.4 — the pictures line up, with a handful of pixels wildly off.

Taken together, those two numbers are the most valuable information available. The shape of the error distribution tells you the class of cause:

DistributionPoints to
Uniform offset, large mean (e.g. 92) A wrong numeric mapping — every sample computed incorrectly
Tiny mean, enormous individual differences Error is spatially local — a branch that only fires in certain places

This was the second shape, so the suspicion should never have been on decode or numeric conversion.

2. The cause was the measurement, not the code

RGBA already contains a decision: chroma upsampling.

In YUV 4:2:0 the chroma resolution is half that of luma (half in each dimension). Converting to RGB means generating a chroma value for every pixel, which requires an interpolation policy:

ImplementationHow chroma is sampled
Our export path The centre sample of each corresponding luma block (nearest-neighbour style)
swscale (ffmpeg) Bilinear interpolation

On flat colour the two agree almost exactly. On colour edges — red text on blue, a saturated highlight, the boundary of a large colour block — they can differ substantially, affecting only the one- or two-pixel rim.

That fully explains "mean < 1.4, maximum 97": spatially local, large in magnitude. Not our bug, not ffmpeg's bug — two different, both defensible implementation choices.

3. The conclusion: change the measurement, not the code

Bypass colour conversion and compare planes. Have ffmpeg emit raw YUV:

ffmpeg -i input.mp4 \
  -vf "select='eq(n\,<display index>)'" -vsync 0 \
  -pix_fmt yuv420p -f rawvideo ref.yuv
ClipCriterionResult
8-bit source Byte-for-byte identical 3110400 / 3110400 bytes identical
10-bit source Bytes differing by more than 1 after shifting by shift + depth - 8 Zero

(3110400 = 1920 × 1080 × 1.5, exactly one 4:2:0 frame.)

This is a much stronger claim than the RGBA attempt: not "close enough" but "byte-for-byte identical" — a criterion you can document and re-run on every future change to the pixel path.

4. Why a difference of 1 is tolerated

The tolerance on 10-bit comes from two roundings: the source is 10-bit, ffmpeg rounds once producing an 8-bit reference, and our path rounds once. Opposite directions give a difference of 1.

That does not relax the standard. Note what changed:

  • Old: "mean difference below some number" — which requires inventing the number, with nothing to base it on;
  • New: "the count of bytes differing by more than 1 must be zero" — exact, enumerable, no room for interpretation.

Conversely, writing "mean < 2" or "difference < 4" would destroy the criterion, because it would also wave through a genuine small systematic error. Tighten the criterion to the upper bound of the rounding error, not to "looks about right".

5. Two more checks the same change needs

1. Chunking invariance

The real export feeds frames in chunks inside a worker (chunking exists so progress and cancellation can be handled between chunks). So replay the worker's actual shape — windowed load plus chunked feed — with chunk sizes of 1 / 3 / 8 / 1000.

The output must be identical every time. Chunking is a control-flow device, not a numeric transform; if changing the chunk size changed the pixels, that would be a real bug. The check is cheap enough that skipping it makes no sense.

2. The message protocol layer

Comparing the C function is not enough. Load the built worker chunk in Node (faking self, importScripts and postMessage) and actually drive the four RPCs: begin / step / finish / cancel. This layer catches things the C layer cannot:

  • whether an old token is genuinely invalidated after cancellation (is the job really released?);
  • whether the number of progress ticks equals the number of chunks;
  • whether the reported picture size is the decoded pixel size;
  • whether the returned bitmap dimensions and RGBA byte count are self-consistent.

This layer did catch something: the assertion "reported size equals the container's track.width" failed, because that field read 1919.593262 — a fixed-point value from tkhd, not a pixel count. Switching to an aspect-ratio comparison (within 1%) was the correct fix. Only the protocol layer exposes this class of problem: the C side computes correctly, and the metadata was never an integer.

6. Transferable lessons

  • Diffing "the thing after conversion" drags the conversion policy into the comparison. To verify an A→B transform, find an intermediate representation that bypasses the extra layers. Here that was YUV planes; elsewhere it may be an intermediate tensor, a pre-serialisation object, or raw unformatted bytes.
  • Read mean and maximum together. Mean alone hides severe local errors; maximum alone mistakes policy differences for bugs. Only the pair reveals the shape of the error.
  • Make criteria enumerable. "Count of bytes differing by more than 1 equals zero" beats "mean below 1.4", because it removes the need to invent a threshold.
  • Test the protocol layer too. Unit-level correctness is not integration correctness. Running the built artefact through its real protocol in Node is cheap and covers argument validation, token lifetime and size self-consistency — things that only fail when assembled.

Related: 10-bit footage renders black · when the browser decoder rejects a stream · How it works · Open the analyzer