As someone else mentioned, the real conference happens around the food, the speakers and presentations are just a side show. If everybody disappeared off with their friends to eat on their own, then you've lost much of the point of a conference. At every conference I've been to the highlights have almost always been meeting awesome people over lunch or drinks, and only very rarely any of the actual presentations.
I've only gone to conference once, but I generally ate out with groups of people who didn't already all know each other. A table only holds so many people, so the same strategy is necessary whether people scatter to restaurants or stay in one place for a catered meal.
Do the attendees really all have to go to the same restaurant?