robots.txt and favicon.ico share nothing in common. Browsers do not automatically request robots.txt every time you load a site. If you count a bot as one visitor, they make up maybe 0.01% of your traffic. It's also an explicit request, the bot actually wants that file. Users don't actually explicitly ask for favicon.ico, their browsers do it for them when they type in pagename.html. It's also not part of the page like the icon is. favicon.ico is like having javascript.js and style.css in your root directory auto-scanned as well.
Unlike favicon.ico, you couldn't just slap a line in your HTML here instead, it would already be too late for crawling. If we tried to put this into the headers for a bot's HEAD request, then people with only file drop access to hosting wouldn't be able to change the robot settings.
That said, I'm always open to considering better ways of doing things. Maybe we could have web servers optionally parse robots.txt locally, and bots could get the information from a special OPTIONS / HEAD request instead.
Humans could also hit such pages. If your GET requests change state, there's no helping you.
How many billions (trillions?) of wasted robots.txt requests have been made because they didn't bother to engineer this the right way and make it a <link> tag in the document?
I wish we could send him the bill for all the wasted power, all the wasted developer time (even ten seconds per developer for all 100 million+ sites out there adds up to quite a lot.) Then see how glad he was he checked this misfeature in."