I use ffmpeg for encoding audio and metadata extraction... very powerful and a really sophisticated tool, but handling timestamps I agree with you. Some caveats that I came across along the way:

- Inaccurate time handling (https://github.com/yermak/AudioBookConverter/issues/21#issue...)

- Incorrect handling of mp3 chapters https://github.com/sandreas/m4b-tool/issues/71#issuecomment-...

If anyone who is interested in using ffmpeg in a docker container (without the dependencies / compiling stuff), this alias is pretty useful (with relative paths ;-):

  alias ffmpeg='docker run --rm -u $(id -u):$(id -g) -v "$PWD:$PWD" -w "$PWD" mwader/static-ffmpeg:4.3.2'