If you have a few fonts, referenced by an external css file, on a page that has a bunch of images, the delay is a couple of seconds on a mobile connection. It's not just the download time for 20kb, it's the servers response time (you _are_ hosting it yourself right?) × number of resources requested.

I'd rather just view the page with the default fonts rather than wait 5-6 secondsfor the minimally different sans serif font that you've chosen for one paragraph that blocks the entire page render.