Performance not competitive against torchcodec

[oakaigh]

It appears that random access (with approximate seeking) in pyav can be 5x slower than torchcodec. The video used for benchmarking is av.datasets.curated("pexels/time-lapse-video-of-sunset-by-the-sea-854400.mp4").

I have tested other mp4 files and for files with short gops (e.g. 2), although the differences are not as big (~2x), seeking seems to consistently take almost the same amount of time as decoding.

[WyattBlue]

The cause is decoder threading, not the seek. av.open() defaults to
thread_type = SLICE, which only parallelizes slices within a frame - and most H.264/H.265 has one slice per frame, so decoding is effectively single-threaded. torchcodec uses frame-level threading.

It hits seeking because seek() only jumps to the nearest keyframe; you then
decode forward to your target. With this file's GOP of 90, that's up to ~89
frames decoded single-threaded per seek — which is why seeking costs about as much as decoding. Short GOPs narrow the gap because there's less to decode
forward.

Fix: opt into frame threading before decoding:

container = av.open(path)
stream = container.streams.video[0]
stream.thread_type = "AUTO"   # or "FRAME"

On my 8-core machine this took 40 random seeks from ~96 ms to ~28 ms each
(3.4x); more cores, bigger win.