Thank you, yes this is my problem which most commenters here are missing.

I'm using Jellyfin server and Infuse on Apple TV, so I don't have a great way to force-slurp a file on first read, at least not without patching Jellyfin. And I'm not super eager to learn C#.

Even if I were streaming to mpv, most of my lan is only gigabit which is much less then what my storage array can put out.