I know when i am doing an interview I am prepared and make sure I know exactly how I am going to structure the interview and have read over the persons resume. I can't say the same for the places I have interviewed.
And on the next day: I spent few hours talking about about programming to programmers. That done, I wondered about the town some more, eating local food and sipping foreign wine.
Damn! What a waste of a holiday.
The next day I had about five or six hours of the toughest interview problems I have ever encountered and they had to end it a slightly early to make sure I had enough time to get back on the airport and make the flight they had scheduled for me to get back. About the only thing I got to experience besides Silicon Valley traffic was a meal at In-N-Out Burger (which was delicious, like always).
I still enjoyed the experience, mind you, and Google campus was really cool to see in person, but that didn't exactly feel like a holiday. I imagine most people's experiences would be more like this and less like your singular holiday-like experience.
There are plenty of non-unicorn places that are awesome to work at. Maybe focus on some startups and other places, you don't have to take the job if they offer you. You are just as much interviewing them to see if it's a good fit.
The other thing too is the more you interview the more comfortable you are with it. You have already experienced probably the tougher interviews out there. Keep at it and you will land somewhere you love.