MP4 MP4 Analyzer by MediaOnDeck

// engineering · fallback

When the browser decoder rejects a stream outright

One HEVC film made Chrome's built-in decoder produce nothing at all, from the first frame to the last, while the same file played fine in other players. Rather than telling the user "unsupported", the tool hands the job to the FFmpeg it already ships — at roughly 9 ms per frame, which is why it comes with progress and cancellation.

1. Symptom: exactly one file with no picture

A user reported it plainly: entering frame analysis raised EncodingError: Decoding error.

Everything else was fine:

  • the box tree was complete and expandable;
  • the frame list enumerated every sample with correct type, size and offset;
  • the binary view showed each frame's raw bytes;
  • other files decoded for preview without issue;
  • the same file played normally in other players.

Together, those narrow the scope sharply: the problem sits between this bitstream and the browser's decoder. Container parsing, the frame tables and our other paths are all uninvolved.

2. Diagnosis: not our code

Frame preview uses WebCodecs VideoDecoder: configure, then feed frames one at a time. This failure was wholesale rejection — not one bad frame, but every frame failing from the start, producing nothing.

The same file decoded perfectly from the ffmpeg command line. So the bitstream is valid and decodable; Chrome's decoder simply will not accept one of its parameters.

Suspicion points at the stream's profile / tier / level (PTL) and sub-layer constraint bits — but that was never confirmed. Recorded honestly: this is an unlocated root cause, not a solved problem. What follows routes around it; it does not fix it.

3. The choice: decode it ourselves rather than call it unsupported

From the user's side, that error message means "your file cannot be used". But the file was fine — the problem was purely with the browser's decoder.

And one thing was already in place: the project ships a wasm module built from FFmpeg (block analysis runs on it). It has full decoding capability; it simply never exposed a path for "the decoded pixels", only MV / QP as JSON.

So we added a pixel-export path and fall back to it when WebCodecs fails. The design goal fits in one sentence: anything FFmpeg can decode can be shown.

4. Implementation: pixel export on the C side

Block analysis walks back from the target frame to an IDR, feeds that byte range to the decoder and decodes frame by frame. Originally, on hitting the target frame it did exactly one thing: serialise that frame's MV / QP into JSON for the JavaScript side.

Now there is a second option, and the two are mutually exclusive:

ModeOn hitting the target framePurpose
Block analysis Serialise MV / QP to JSON QP heatmap and motion-vector arrows
Frame preview Pack the pixel planes The picture itself; the JSON is useless here

The "is this the target frame" test is identical in both, so making them exclusive is safe. The reason to make them exclusive is that JSON serialisation is not cheap and preview never needs it.

Packing is driven by the pixel format descriptor: strip the row stride from each of the three planes and lay them out as one contiguous Y | U | V block. Because the loop follows the descriptor rather than switching on format names, yuv420p, yuv422p, yuv444p, nv12, p010le and yuv420p10le share a single packing routine.

Two easy traps. First, converting a 16-bit container to 8-bit requires a shift of shift + depth - 8, never a hardcoded 8 — that trap has its own note. Second, the "capture pixels" switch must be enabled after the job begins, because the begin step clears it. Get the order wrong and nothing is captured, silently.

5. The JavaScript side: why colour conversion lives there

This wasm is built without a colour conversion library — meaning there is no colour converter inside it. So YUV → RGBA happens in JavaScript:

  • Downscaling uses box averaging — averaging the samples inside each source block. At preview sizes this is visibly cleaner than nearest-neighbour, especially at larger reduction ratios;
  • the colour matrix comes from the container's colour space (BT.601 / 709 / 2020), with limited versus full range handled separately. Choosing the wrong matrix shows up as a global colour cast with perfectly correct structure.

This also states a boundary: the path handles planar and semi-planar YUV only. Other formats need their own conversion.

6. Why the feed is chunked

The most misread part of this whole thing: chunking is not about memory.

Before decoding each frame, the C side reuses a single frame object, so memory does not accumulate with the number of frames fed. Feeding the whole chain at once and feeding it in ten pieces have nearly identical peak memory.

Chunking exists so the worker can handle progress reporting and cancellation messages between chunks. Feeding the entire chain in one call blocks the worker for up to several seconds — the progress bar never moves and the cancel button does nothing, so the UI appears frozen.

StepPurpose
1. Set upWalk back, load the window; returns the total frame count
2. FeedFeed N frames; returns frames fed and total
3. FinishFeed the remainder, export pixels, return RGBA data
4. CancelAbandon the job and release resources

Alongside it is a token mechanism: one token per job, and once cancelled, later requests carrying the old token are rejected. This guards against a race where the user switches frames and cancels, but the old job keeps running — finishing after the new picture arrives and overwriting it.

7. The cost: slow, but it is a fallback

MetricValue
Software decode per frame≈ 9.0 ms
Worst case (target 250 frames past an IDR)≈ 2.2 s
WebCodecs hardware decodeFar faster — that is the fast path, not the one being replaced
Byte-level safety valve256 MB
  • WebCodecs is attempted first; the fallback runs only on failure or unavailability;
  • the tool remembers, per track, that a track cannot be decoded this way, so later frames skip a doomed attempt instead of waiting through it;
  • if the resolved window exceeds 256 MB it degrades back to the original error message rather than consuming all the memory.

8. Transferable lessons

  • Classify platform failures. "Our code is wrong" and "the platform will not accept this input" demand different responses: fix versus fall back. Conflating them leads to long hours spent trying to fix something unfixable.
  • A fallback needs progress and cancellation. Users tolerate "slow by two seconds"; they do not tolerate "frozen for two seconds with no indication of why". Chunking serves that, not memory.
  • A fallback must not share assumptions with the primary path. If the fallback depends on the same decoder, it cannot fall back from anything. This path works precisely because it uses a completely different implementation: FFmpeg software decode versus platform hardware decode.
  • Write down root causes you have not located. "Suspected PTL constraint bits" is not a conclusion. Stating clearly that it remains unlocated is far more useful than vaguely implying it is fixed — the next person, including future you, needs to know where to pick up.

Related: verifying pixel export on YUV planes · 10-bit footage renders black · Guide · Open the analyzer