MP4 MP4 Analyzer by MediaOnDeck

// engineering · bitstream

623 keyframes, zero detected

The container declared 623 sync samples. The code recognised none of them, and block analysis on a 1.7 GB film came back empty every single time. The cause lived inside a very natural assumption: that the first NAL unit in a sample tells you what the frame is.

1. Symptom: a healthy frame list, empty analyses

The file opened fine. The box tree was complete, the frame list enumerated all 379,000 samples, and the parameter sets came out of avcC without complaint. But clicking block analysis on any frame produced one of two outcomes: an empty result, or the error "GOP too long".

Those look like the same problem. They are two different failure paths — worth remembering.

The first move was to narrow the scope: take every sample listed in the stss box and try them one by one. All 623 failed. That number was the most valuable clue available, because an intermittent parsing bug does not fail 100% of the time. Only a condition that is systematically false does.

2. Why block analysis has to walk back to an IDR

Inter prediction in H.264 and H.265 means P and B frames cannot be decoded independently — they depend on reference frames already decoded. To analyse frame N you must start from the nearest preceding IDR and feed every frame in between, in order. 

target frame
  └─ walk backwards to the nearest IDR
       └─ feed [start of IDR, end of target frame] as one byte range
            └─ decode frame by frame until the target

The walk-back depends entirely on frame type detection. Get that wrong and you land on one of two very different failures:

How detection failsBehaviourSymptom
Every frame judged non-IDR Walks back until it exceeds the safety bound "GOP too long"
Starts feeding from a non-IDR The decoder never gets its reference, emits nothing Always-empty result

Two paths, one symptom — and neither of them looks like "we misread the NAL type". They look like "this file is broken". Which is why this bug survived so long.

3. Root cause: reading only the first NAL unit

The original implementation read the sample's bytes, parsed the first NAL unit, and declared the frame an IDR if its type was 5 (H.264 IDR slice).

The assumption looks unassailable until you print the actual data:

sample #0    NAL chain: 6 (SEI), 6 (SEI), 5 (IDR)
sample #250  NAL chain: 6 (SEI), 6 (SEI), 5 (IDR)
sample #500  NAL chain: 6 (SEI), 6 (SEI), 5 (IDR)

x264 prefixes each IDR sample with two SEI messages (type 6, supplemental enhancement information). The first NAL type therefore never needs to be 5, and the comparison never fires once. All 623 IDRs, missed.

This bug has an unhelpful property: it only affects encoder output that carries a SEI prefix. A different file might work perfectly. So it is neither reliably reproducible nor random — it succeeds or fails wholesale, per source file. That is worse than intermittent.

4. The correct rule: scan the chain, take the first VCL

NAL units fall into two groups: VCL (Video Coding Layer — the actual slice data) and non-VCL (parameter sets, SEI, access unit delimiters, and so on). To classify a frame, skip every non-VCL and look at the first VCL unit.

StandardVCL rangeIDRCommon non-VCL
H.264 1 – 5 5 6 SEI · 7 SPS · 8 PPS · 9 AUD
H.265 0 – 31 19 / 20 32 VPS · 33 SPS · 34 PPS · 35 AUD · 39/40 SEI

Walking the chain itself has two variants, depending on how the bitstream is packaged: Annex B splits on 00 00 01 start codes, while samples inside MP4 use length prefixes whose byte width comes from lengthSizeMinusOne in avcC / hvcC (usually 4 bytes). 

scan the NAL chain inside the sample:
  read the next NAL type nType
  if nType is VCL:              # H.264: 1..5   H.265: 0..31
      frame type = nType
      break                     # first VCL only; later slices belong to the same frame
  else:
      advance to the next NAL    # skip SEI / SPS / PPS / AUD

One more mistake worth recording: the original predicate was "type <= 9, or anywhere in 16..21, means IDR". That range is wrong for both standards — for H.264 it sweeps SEI/SPS/PPS/AUD into the VCL set, and for H.265 it straddles BLA, IDR and CRA, three distinct kinds of random access point. If you are about to write a magic range, open the standard's table first.

5. Why the container's stss is not enough on its own

MP4's stss box lists sync samples — random access points. It looks like the answer is handed to you. Two reasons it is not:

  • It is a container concept, not a coding one. In HEVC, CRA (Clean Random Access, NAL type 16) is also a legitimate random access point, but it is not an IDR — pictures decoded after a CRA may still reference content from before it. For the question "where can I start feeding", CRA and IDR are not interchangeable.
  • It gives sample numbers, not byte semantics. Walking back needs a file offset to start feeding from, and converting sample index to offset also depends on the sample tables.

So stss was used for cross-validation: if the IDR set derived from the bitstream shares nothing with the declared sync samples, the detection code is broken — fail loudly rather than degrade quietly.

6. How the fix was proven

  1. Coverage. All 623 sync samples from stss produce a non-empty block analysis. Before: 0 / 623. After: 623 / 623.
  2. Alignment with native truth. Build a native keyframe list with ffprobe -show_entries frame=key_frame and compare per frame against the start point we walked back to. Our start point must never be later than the native IDR — that would mean dropped frames.
  3. Numeric agreement. On a 1634 MB clip, motion-vector and QP statistics before and after the change both agree with native ffprobe -export_side_data mvs output. This also covers "we did not break anything else".

One useful side effect: after the fix, the average GOP window on that 1.7 GB film is 0.46 MB. Walking back no longer hits a bound, and the load stays constant-size — covered in the large-file note.

7. Transferable lessons

  • "Read the first X and conclude" is nearly always wrong. A bitstream may carry any number of any kind of non-VCL NAL before the VCL one, in no order you control. Print the real data before trusting any "first" or "at the front" assumption.
  • A 100% failure rate is a gift. It collapses the search space from "the whole decode path" to "one condition that is always false". Conversely, when the failure rate is not 100%, the first question is what the passing samples have in common.
  • A predicate needs an independent source of truth. Here it was ffprobe's keyframe flag. Validating a predicate with the same code that produced it validates nothing.

Related: motion-vector ground truth with ffprobe · How it works · Open the analyzer