MP4 MP4 Analyzer by MediaOnDeck

// engineering

Engineering notes

Real problems hit while building this tool, and how each fix was actually proven. Bitstream, memory and pixel-level write-ups — including the cases that looked like an FFmpeg bug and turned out to be ours.

1. Why these are written down

Debugging video coding has a particular shape: failures all look alike, but the causes are nowhere near each other. An empty block analysis, a black frame, a file that always reports "GOP too long" — any of those can come from a misread NAL type, from a single shift amount, or from a GB-scale file that does not fit in a 2 GB heap. Symptoms alone lead you the wrong way.

So every note follows the same structure: symptom → how the cause was isolated → what changed → what evidence proved it. The last part is usually the useful one, because "it looks right now" and "it is right" are different claims.

2. The notes

623 keyframes, zero detected

The container declared 623 sync samples; the code recognised none of them, and every analysis came back empty. The cause was the assumption that the first NAL in a sample tells you the frame type — x264's actual output is [SEI, SEI, IDR]. Includes the H.264 and H.265 NAL type tables, and why stss alone is not enough.

10-bit footage renders black: one shift amount

8-bit sources exported byte-for-byte correct; a 10-bit source came out entirely black (mean difference ≈ 92, the signature of a flat black frame). The cause: always shifting the 16-bit container right by 8, when P010LE and yuv420p10le align their bits differently.

Fitting a GB-scale file into a 2 GB wasm heap

A 1.7 GB film has to be parsed inside WebAssembly linear memory. The approach is to load only the byte window covering the target GOP — measured at 0.46 MB. Plus three things that really did blow up: teardown frees the file buffer, the heap only ever grows, and why a frame-count safety valve is the wrong shape.

Building motion-vector ground truth with ffprobe

When you suspect your MV export is broken, one ffprobe -export_side_data mvs produces native ground truth. The result: "frames with no motion vectors" is a legitimate outcome, and our output matched natively, frame for frame. Same family of trick: display order ≠ decode order, so look it up via pkt_pos.

Why pixel export has to be verified on YUV planes

Comparing RGBA showed per-pixel differences up to 97 of 255 on 10-bit material — a "serious bug" — while the whole-frame mean difference stayed under 1.4. The reason is that RGBA already contains a chroma upsampling policy. Comparing planes instead: 3110400 / 3110400 bytes identical on 8-bit, and zero bytes differing by more than 1 on 10-bit.

When the browser decoder rejects a stream outright

One HEVC stream made Chrome's WebCodecs decoder fail on every frame, while other players handled it fine. Rather than telling the user "unsupported", the tool falls back to its bundled software decoder — at roughly 9 ms per frame, which is why it ships with progress and cancellation.

One skipped bit field shifted the next 32 bits

The parsed chroma format came out as 655392 against a legal range of 0 to 3. One reserved group inside a parameter-set structure had been skipped, shifting every later field by 32 bits. Includes why your own parser can never be its own ground truth.

269 of 271 frames had the wrong reference list

MP4 samples split NAL units by length prefix — there are no start codes at all. Flip the detection order and every NAL header is read from the wrong offset: 269 of 271 frames disagreed with ground truth, with no error raised.

Moving 92% of a file to read four bytes

Reading each frame's I / P / B type once meant reading the whole sample: 1506 MB moved across a 1.7 GB film, in 6.9 seconds. A grow-on-demand byte window took it to 0.84% and 460 ms, with byte-identical frame types.

3. Every claim comes with a way to check it

The numbers in these notes are reproducible, not impressions. Four techniques recur across all of them:

Technique What it settles
Compare planes, not RGBA Separates "different colour-conversion policy" from "broken export", yielding byte-level evidence
Build ground truth with native tools first Before asserting "this is our bug", diff against ffmpeg / ffprobe output — usually one command is enough
Windowed vs whole-file A/B Proves an equivalent transform really is equivalent, instead of assuming it
Measure, don't infer To know whether a path reads large files, add a byte counter and run it once — inferring from data structures gives the opposite answer

4. Known boundaries

Two of these are still not fully understood. Recorded honestly so they are not mistaken for solved:

  • The root cause of Chrome rejecting that HEVC stream is still unknown. Suspicion points at the stream's profile / tier / level sub-layer constraint flags, but it was never confirmed. It is no longer exposed in the product — you see the picture, just via the software path.
  • Files over 1 GB are infeasible on phones. That is a single-process memory budget, not a performance problem to optimise. Desktop is unaffected.

5. Related pages

  • How it works — the overall architecture: three layers and the data flow.
  • Guide — the end-to-end workflow, from opening a file to reading QP and MV overlays.
  • About — project background, verification methodology and the full list of limitations.
  • Open the analyzer — reproduce any of the above with your own files.