Perhaps you just need to increase the smoothing parameter. I found the default of 10 way too low, and needed around 50-100 for my very shaky 50 fps home videos.
See here: https://ffmpeg.org/ffmpeg-filters.html#vidstabdetect-1
Maybe your ffmpeg build doesn’t have it? Also try removing the log level parameter.
>which is then used by the vidstabtransform filter.
The first pass is meant to find the rotation, then I assume the next pass cancels it out.
https://www.youtube.com/watch?v=I6E6InIQ76Q&t=9s
They stay stable in the center whereas the rest of the frame is transformed. This necessarily requires introducing black/white crop frames into the image with all kinds of shapes and sizes, but it turns out incredibly smooth and doesn't lose any of the frame. It also requires no noticeable blurring at all from what I can tell. But that doesn't seem to be quite what's happening with these commands though.
I was able to find this comment thread where a user posted some ffmpeg scripts to replicate the behavior. Am on mobile currently so I can't verify they do what you're looking for but here's a snippet.
//PART 1 [Defaults: shakiness=5:accuracy=15:stepsize=6:mincontrast=0.3:show=0] ffmpeg -i shaky-input.mp4 -vf vidstabdetect=shakiness=5:accuracy=15:stepsize=6:mincontrast=0.3:show=2 dummy_crop.mp4
//PART 2 ffmpeg -i shaky-input.mp4 -vf scale=trunc((iw0.90)/2)2:trunc(ow/a/2)*2 scaled_crop.mp4
//PART 3 [-strict -2 ONLY IF OPUS AUDIO] - [Unsharp Default: '5:5:1.0:5:5:0.0'] ffmpeg -i scaled_crop.mp4 -vf vidstabtransform=smoothing=20:input="transforms.trf":interpol=no:zoom=-10:optzoom=2,unsharp=5:5:0.8:3:3:0.4 stabilized_crop-output.mp4
https://www.reddit.com/r/stabbot/comments/9f7ayj/comment/e5x...
I don't think vidstab can do exactly what's being asked for here, I've never seen it rotating the frame completely to keep a rotating pen stabilized, or move the frame inside a larger canvas to keep a moving subject centered. I think you have to use video editing and do that manually.
Update: Just tried it. It seems to do some kind of blurring to smooth the vibrations over several frames instead of trying to cancel the shaking by inverse-transforming the frame with respect to the shake. I think I've tried this in my experiments before -- and I guess it's the difference between "de-shake" and "stabilization"? Sadly it doesn't really seem better to my eyes sadly, but thanks for the help anyway.