I've run a very small (~20) conference over the last 15 yrs and this is basically the plan. We even had it on a cruise ship one year. The friendships, and collaborations that have ensued are a testament to its effectiveness.

(To be honest we do have a few short talks and an agenda to guide the discussions, but these are mostly requirements of the funding that we've found)