Every Matrix protocol client I've ever tried, including yours, seems to connect to your "strictly optional" centralized directory service. I've never once wanted to use it, and, despite your claims of it being optional, 100% of the times I have logged in to a Matrix instance, via the web or via an app, my computer has talked to your computer.

Something's broken here.