Having plugins installed has no performance incursion at all (because they don't run unless configured), but using them will, of course affect performance. But it depends on the plugin and what it does.

You have to do your own benchmarking to see what effect they have to performance. Everyone has different needs and configs, so it's not possible to give authoritative benchmarks. Just try it out and see.

Regarding brotli, there is a plugin you can try if you'd like https://github.com/ueffel/caddy-brotli, but brotli's on-the-fly performance is not that good. It works well if you pre-compress you assets though, and you can serve those with the file_server directive's precompressed option.