I typed "sushi" on duckduckgo.com and saw a result from Yelp. If duckduckgo.com used Yelp's API to get that result then you can imagine the complexity of maintaining API connections to hundreds of services like Yelp that do not allow crawling. However if duckduckgo.com crawled/scraped Yelp to get that result then that's against Yelp's TOS which would cause Yelp to send a cease and desist order to duckduckgo.com.
IMHO any crawler that drives traffic back to the original site (like Google, Bing, duckduckgo, etc) should be allowed to crawl the site but Yelp's TOS prevents that.
I used to work at Yelp, and I can tell you that if there were no restrictions on who crawled the site, we'd have been regularly DOSed by robots. I don't (and didn't) like the precedent, but I understand some of the motivation.
If they catch you crawling, they'll send you a cease and desist letter, so do it at your own risk.
Plus, the privacy promotion from them helped drive developers to their sites.
Restrictions on what people can do on the internet is a slippery slope.
You're not going to crawl / index better than Google.
Maybe you could combine the DMOZ directory and Google results somehow.
Rather than replacing Google, try to supplement their services or just somehow provide more value to users.