If you accidentally put a link on your public website to content you didn't want public, and Google indexes it, then someone else fetches it, yeah, that's your fault for exposing it. It's also your responsibility for telling Google to remove it from their index. But if someone is poking around to see if urls are valid (eg: the /.ssh/id_rsa requests my website gets) that's breaking and entering, even though I'd be negligent for leaving my keys on the webserver like that.
I imagine in addition to public URLs they also use URLs entered into Chrome (which isn't that public) and since they host DNS they probably crawl domain names requested through them too.
Legally I'm not sure what they stand on to do that but they're big enough now so I guess they don't worry too much about that nowadays.
For example, in the weeks before an Apple event, people tend to visit presumed unannounced product pages (by guessing) etc.
The question is of course in what situations I should have known I stumbled across something they didn't intend for me to see.
Googles scraper certainly walks up to every door, and then publishes what it finds. If you accidentally publish something and then google index it, and someone reaches it via google, it's hard to say it wasn't public.