This. When optimizing realtime apps, focus on worst cases.
Also, FPS is inconvenient metric because framerate is not a linear scale. You can't say a task takes x FPS to compute, but you can say it take x milliseconds, since time is a linear scale.
For example: if a task takes 10 ms and other stuff takes 1 ms. FPS delta for the task is ~909 FPS (1000 FPS - 91 FPS). But if other stuff takes 20 ms, we get only a ~17 FPS difference (50 FPS - 33 FPS).
For example a game which runs at 120 fps but every 10 seconds has 1 frame that takes 1/30th of a seconds feels awful.
A game that runs at constant 60 fps feels much better.
In this case it's better to just count the frames that took too long and by how much.