// engineering · methodology

用 ffprobe 建运动矢量真值，别急着啃源码

发现某些帧「没有运动矢量」时，第一反应是怀疑自己打补丁重新编译的 FFmpeg 出了问题。正确的下一步不是打开解码器源码，而是先建一份原生真值——一条命令，三十秒结案。

一、症状：一批帧「没有运动矢量」

块分析能导出每一帧的运动矢量（MV）。在某个 1634 MB 的 H.264 素材上，导出结果里有若干帧几乎没有任何箭头。

这个怀疑很合理：块分析用的是本地打过补丁、重新编译的 FFmpeg（为了把 MV 与 QP 导出到 JS 层），所以「是我们的改动搞坏了」是一个自然的猜想。但沿着这个猜想走下去，就是要读解码器源码去核对 MV side data 的填充逻辑——那是几小时量级的活，而且很可能得出和下面一样的结果。

所以先做另一件事：产出一份不依赖我们任何代码的原生输出。

二、一条命令建真值

ffprobe -export_side_data mvs -show_frames input.mp4

-export_side_data mvs 让 ffprobe 把解码过程中得到的运动矢量作为 side data 附加到每一帧上输出。输出量很大，配合 -show_entries 裁剪才有实用价值：

ffprobe -v error -select_streams v:0 \
  -export_side_data mvs \
  -show_entries frame=pkt_pts_time,key_frame:frame_side_data \
  -of csv=p=0 input.mp4

结果：

素材	范围	没有 MV 的帧
1634 MB H.264	前 250 帧	第 0 帧（I 帧）与第 150 帧

我们的导出结果与它逐帧一致——包括那两帧。没有 bug。整个排查大约三十秒。

三、为什么会「合理地」出现没有 MV 的帧

这是这次排查最有价值的收获：「这一帧没有运动矢量」是一个合法结果，不是错误。

至少有两种正常情况：

帧内编码的帧（I 帧）。它根本不参与帧间预测，自然没有运动矢量。上表第 0 帧属于这一类。
名义上是 P / B 帧，但所有块都选了帧内预测。典型触发是场景切换：新的画面和参考帧完全对不上，编码器判断「全部重画」比「找一个匹配块」更划算。上表第 150 帧属于这一类—— 至于是场景切换还是别的编码器决策，从「MV 缺失」本身看不出来，要看该帧 slice 里的预测模式分布。

由此得出一条要划掉的假设：「帧类型是 P / B，就必须有运动矢量」是错的。 任何以它为前置条件的检查，都会在场景切换处误报。

把这条反过来用：如果某个 P / B 帧完全没有 MV，它值得看一眼，但不构成错误。真正该报错的是「有 MV，但参考帧索引指向了不存在的帧」这类结构性矛盾。

四、同源技巧：显示序 ≠ 解码序

另一个反复踩的坑与 ffprobe 有关，只是不在 MV 上：B 帧的存在意味着解码顺序与显示顺序不同。

想从 ffmpeg 里单独取某一帧做对拍时，select='eq(n,K)' 里的 K 是 解码 / 输出顺序的下标，不是你在工具界面里看到的那个帧号。直接拿界面上的帧号填进去，取到的是另一帧，然后你会开始怀疑自己的解码结果。

做法是用包偏移反查：

# 建「显示序 → 包偏移」表
ffprobe -v error -select_streams v:0 \
  -show_entries frame=pkt_pos -of csv=p=0 input.mp4

# 拿目标样本的绝对字节偏移去这张表里找下标
# → 得到 select='eq(n\,下标)' 里该填的数字

实测两个例子，差距远超「差一帧」的量级：

素材	工具里的 sampleIndex	ffmpeg 的显示序
片子 A	33	32
片子 B	135	269

差 134 帧。靠猜 POC、靠「大概差几帧」都是没戏的——必须用字节偏移反查。

顺带说，这类判据还要做一次反向对照，证明它有区分度：把导出的画面与 ffmpeg 的 相邻显示序帧比，如果判据有效，正确帧应该明显更接近：

比较对象	平均差	最大差
正确帧	0.14	1
相邻显示序帧	3.17 – 7.83	255

差两个数量级，判据可信。否则「对上了」可能只是「随便两帧都很像」。

五、两个会白白吃掉半小时的小坑

尾随逗号。ffprobe -of csv=p=0 输出的行可能带一个尾随逗号（例如 44,）。直接用 Number('44,') 得到 NaN，然后这条记录被静默过滤掉——你会以为「这个偏移在 ffprobe 里根本找不到」，进而怀疑帧号换算错了。取值时先按逗号切分再转数字。
maxBuffer。在 15.5 万帧的轨道上，ffprobe 的输出会超过 Node execFileSync 默认的 1 MB 缓冲上限，抛 ENOBUFS。这跟你分析的文件毫无关系，纯粹是测试脚手架的默认值不够。把 maxBuffer 调大即可。

这两个坑的共同点是：症状看起来像「被测对象有问题」，实际是「测量工具没配好」。 把测量链路的每一步都当作可疑对象，是这类排查里最省时间的一条纪律。

六、方法学：断言「是我们的问题」之前先建真值

这次的复盘结论很直接：

路径	成本	结果
读 FFmpeg 解码器源码找 MV 填充逻辑	几小时	大概会得到同样结论
用 ffprobe 建原生真值	约 30 秒	确定性结案

于是把它固定成一条流程：

先问「原生工具能不能产出等价的量」。能，就先生成一份，再对比。不能，再考虑读源码。
真值不能来自被验证的代码路径。用同一份实现既做产出又做验证，等于什么都没验证。
结论要能区分「我们错了」与「本来就是这样」。这次属于后者，而它看起来和前者一模一样。

这条流程有前提：原生工具得真的具备等价能力。运动矢量恰好有现成的导出（mvs）； QP 就没有，所以 QP 那一侧仍然靠别的办法——把帧内 / 帧间块的统计量与原生解码器的输出对拍，而不是直接比数值。

七、可迁移的经验

能用现成工具建真值时，不要读源码。读源码是最后手段，不是第一反应。
「某个量是空的」先问「它可能是合法的空吗」。这一问几乎不花时间，但能省掉一整轮错误方向的排查。
判断帧身份永远用字节偏移反查，不要猜顺序。解码序与显示序的差距可以到上百帧。
判据本身也要验证。做一个反向对照，证明「正确的那一个」确实比「相邻的那一个」更接近，否则「对上了」没有信息量。

// engineering · methodology

Building motion-vector ground truth with ffprobe

When some frames appeared to have "no motion vectors", the first instinct was to suspect the locally patched FFmpeg build. The right next step was not to open the decoder source, but to generate native ground truth — one command, thirty seconds, case closed.

1. Symptom: a batch of frames with "no motion vectors"

Block analysis exports the motion vectors (MV) of every frame. On a 1634 MB H.264 clip, a number of frames in the export carried almost no arrows at all.

The suspicion was reasonable: block analysis runs on a locally patched, locally rebuilt FFmpeg (the patch is what exports MV and QP to the JavaScript layer), so "our change broke something" is a natural hypothesis. Following it, though, means reading decoder source to check how MV side data is populated — hours of work that would very likely have reached the same conclusion as below.

So we did the other thing first: produce output that does not depend on any of our code.

2. One command builds ground truth

ffprobe -export_side_data mvs -show_frames input.mp4

-export_side_data mvs makes ffprobe attach the motion vectors obtained during decoding to each frame as side data. The output is enormous, so trim it with -show_entries:

ffprobe -v error -select_streams v:0 \
  -export_side_data mvs \
  -show_entries frame=pkt_pts_time,key_frame:frame_side_data \
  -of csv=p=0 input.mp4

The result:

Clip	Range	Frames with no MV
1634 MB H.264	First 250 frames	Frame 0 (an I-frame) and frame 150

Our export matched it frame for frame — including those two. No bug. The whole investigation took about thirty seconds.

3. Why frames legitimately can have no motion vectors

This was the most valuable outcome: "this frame has no motion vectors" is a legitimate result, not an error.

Intra-coded frames (I-frames). They do not participate in inter prediction at all, so they have no motion vectors. Frame 0 above is one of these.
Nominally P or B frames where every block chose intra prediction. A typical trigger is a scene cut: the new picture has nothing to match against, and the encoder decides redrawing the whole frame is cheaper than finding a match. Frame 150 is one of these — whether it was a scene cut or another encoder decision cannot be told from the missing MVs alone; you would look at the prediction-mode distribution in that frame's slices.

Which retires an assumption worth crossing out: "if the frame type is P or B, it must have motion vectors" is false. Any check built on it will misfire at scene cuts.

Inverted: a P or B frame with no MVs at all is worth a look, but it is not an error. What genuinely deserves an error is a structural contradiction — for example a frame that has MVs whose reference index points at a frame that does not exist.

4. A sibling technique: display order ≠ decode order

Another recurring trap involves ffprobe, just not the MV side of it: the presence of B-frames means decode order differs from display order.

When extracting a single frame from ffmpeg for a comparison, the K in select='eq(n,K)' is a decode / output order index, not the frame number you see in a tool's interface. Feeding in the interface number extracts a different frame — and then you start doubting your own decoder.

Look it up by packet offset instead:

# build a "display order → packet offset" table
ffprobe -v error -select_streams v:0 \
  -show_entries frame=pkt_pos -of csv=p=0 input.mp4

# then find the target sample's absolute byte offset in that table
# → that index is what goes into select='eq(n\,index)'

Two measured examples, both far beyond "off by one":

Clip	sampleIndex in the tool	Display index in ffmpeg
Clip A	33	32
Clip B	135	269

A gap of 134 frames. Guessing POC, or "roughly a few frames off", does not work — resolve it by byte offset.

Any such criterion also needs a reverse control to show it discriminates. Compare the exported picture against ffmpeg's neighbouring display-order frame; if the criterion works, the correct frame should be clearly closer:

Compared against	Mean diff	Max diff
The correct frame	0.14	1
Neighbouring display-order frame	3.17 – 7.83	255

Two orders of magnitude apart, so the criterion is trustworthy. Otherwise "it matches" may just mean "any two nearby frames look alike".

5. Two small traps that cost half an hour each

Trailing commas. Lines from ffprobe -of csv=p=0 can end with a comma (for instance 44,). Number('44,') is NaN, the record gets silently filtered out, and you conclude "that offset isn't in ffprobe's output at all" — then start doubting your frame-index conversion. Split on the comma before parsing.
maxBuffer. On a 155,000-frame track, ffprobe's output exceeds Node execFileSync's default 1 MB buffer and throws ENOBUFS. This has nothing to do with the file under analysis; it is purely a test-harness default. Raise maxBuffer.

Both traps share a shape: the symptom looks like "the thing under test is broken", when in fact "the measuring tool is misconfigured". Treating every link in the measurement chain as suspect is the cheapest discipline available in this kind of investigation.

6. Methodology: build ground truth before asserting "this is our bug"

Path	Cost	Outcome
Read FFmpeg decoder source for MV population	Hours	Probably the same conclusion
Build native ground truth with ffprobe	≈ 30 s	Definitive

So it became a standing procedure:

Ask first whether a native tool can produce an equivalent quantity. If it can, generate one and compare. Only if it cannot should reading source be on the table.
Ground truth must not come from the code being verified. Producing and validating with the same implementation verifies nothing.
The conclusion must distinguish "we got it wrong" from "it is like that anyway". This case was the latter, and it looked exactly like the former.

The procedure has a precondition: the native tool must genuinely be equivalent. Motion vectors happen to have an exporter (mvs); QP does not, so that side still relies on other means — comparing intra/inter block statistics against the native decoder's output rather than diffing numbers directly.

7. Transferable lessons

When an existing tool can give you ground truth, do not read source. Source reading is the last resort, not the first reflex.
When a quantity comes back empty, ask whether empty is legitimate. The question costs nothing and can save an entire wrong-direction investigation.
Resolve frame identity by byte offset, never by guessing order. The gap between decode and display order can be over a hundred frames.
Validate the criterion itself. Run a reverse control showing the correct frame is genuinely closer than its neighbour, otherwise "it matches" carries no information.