Maybe an area you want to carve out an exception for are benchmarking tests; i.e. tests that ensure that a function runs faster/slower than a certain speed. In this case, you probably need more than 100ms of testing for accurate results.

For example, password checking functions should be slow, and extremely hot/performance sensitive functions should be fast.

Of course, you should have fairly wide margins to avoid flakes.