What are the downsides, load time aside?

Besides just the load time, there’s the “pop” effect of the text not being visible until the font loads (or slightly better but still not great, the page shifting around as default fonts get subbed out for your custom font with different metrics).

On macOS and iOS specifically, the system font (San Francisco) has had a great deal of time and effort poured into readability — so much that rather than being a singular font, it’s a massive family of fonts or metafont with countless variants for various DPIs and use cases. For example, the “San Francisco” used for headings (h1, h2, etc) uses wider glyphs and more relaxed kerning, as is appropriate for headings and the “San Francisco” used for small text uses a variant that has its geometry tweaked to be more legible. It’s a free readability boost that all happens automatically when you use system-ui or -apple-system in your font stack. By using a custom font, you forfeit all of that.

The pop effect can be tamed with, for example, font-display: optional;, which allows the first page load to use the fallback fonts.