Caddy's performance is competitive with nginx, but there are too many dimensions to discuss here. I guarantee that for most people, your web server will not be your bottleneck.

Dynamic brotli is expensive, as brotli compression is inefficient. There are no optimized Go implementations of this yet. But Caddy can serve precompressed brotli files without any separate plugins.

What do you mean by "what affect do [plugins] have on performance"? All features in Caddy are plugins, some just come installed by default: https://caddyserver.com/docs/architecture -- even its HTTP server is a plugin. (All plugins are compiled in natively.)