> set the start media timestamp to the frame that you actually wanted to cut at

You can do that, but it’s container format-dependent (e.g. mov edit list) and player support is uneven. If the player doesn’t support it you get extended cut and A/V desync.