That's a poor example, because they most likely did not intend for that information to be publicly available, and just did a poor job securing it.

Real-world analogy: if you happen to find someone's house with an unlocked front door, that does not give you permission to go looking around inside. It doesn't matter that you don't steal anything, and it doesn't matter that nobody knows you were there. You're still not allowed in and by going in you're breaking the law.

If your example was for content that's retrieved as the default web page for a public DNS entry, or a page that's reachable by following links from the website's default page (including urls in meta, link, and script tags), then I think you're on solid standing as far as being allowed to retrieve the content. At that point, copyright laws and ToS govern what you're allowed to do with the content, with the safe assumption being that you're allowed to view it, but not republish it (except for fair-use excerpts.)

But in this case you are asking the server for permission to come in. Unlike a lock, a server has situational awareness to decide if it says yes or no. We even call it a server, à la servant. If the maid acting as an agent for the homeowner at this hypothetical house invited you in, against the wishes of the homeowner, are you still breaking the law?

Your example is more akin to the maid wandering off leaving the front door open. That's not an invite to come in, that's a poor choice of maid.

We can agree that a maid that lets the wrong people in is a poor choice of made, but the maid is the one responding to the request. A door/lock does not deal with requests at all.

To extend this analogy a little bit, lets say that the homeowner told the maid that they can "give the old tv, the toaster or any other item in the garage to anyone who asks." A stranger comes and asks the maid for the golf clubs in the garage that the homeowner forgot about.

In this case, the homeowner did not explicitly expose the golf clubs to the public, but gave permission to the maid to give them out. Did the homeowner not grant public access to the golf clubs?

The question is of course in what situations I should have known I stumbled across something they didn't intend for me to see.

Googles scraper certainly walks up to every door, and then publishes what it finds. If you accidentally publish something and then google index it, and someone reaches it via google, it's hard to say it wasn't public.

If you accidentally put a link on your public website to content you didn't want public, and Google indexes it, then someone else fetches it, yeah, that's your fault for exposing it. It's also your responsibility for telling Google to remove it from their index. But if someone is poking around to see if urls are valid (eg: the /.ssh/id_rsa requests my website gets) that's breaking and entering, even though I'd be negligent for leaving my keys on the webserver like that.

I imagine in addition to public URLs they also use URLs entered into Chrome (which isn't that public) and since they host DNS they probably crawl domain names requested through them too.

Legally I'm not sure what they stand on to do that but they're big enough now so I guess they don't worry too much about that nowadays.

For example, in the weeks before an Apple event, people tend to visit presumed unannounced product pages (by guessing) etc.

I'm guessing that varies and is decided by legal precedent only once it becomes an issue.

But my point is this: what service have I used, and what terms can I be assumed to have agreed to if I do this

    > wget $random_url > /dev/null

In this case it thus doesn't whether or not you even saw the contract. You are bound by the ToS because it can be expected of you to search for it during continued use of the service.

For example, it is illegal for companies to document the religious beliefs of a person BUT if this is a service for certain religious community (e.g. a forum for a church) then the term that they have to disclose their religious belief is reasonable (given that they don't share this information with anyone, of course).

As for the ads, I don't know. It might be unreasonable to have users agree to subject themselves to third-party ads in order to use the service. But again, I am not a laywer.

I'd say the TOS have to apply to the "service" you're consuming and not the website itself. Otherwise, what happens if you use just a text based browser? Or any other means not foreseen by the TOS?

An ad-blocker would be just the same, it's something that your browser does.

However, in other countries such as the US contract law is totally different, so this does not generalize at all.

Also, IANAL.

That material has been reduced to a number - which certainly can't be considered protected as a material.

Also disappointing that you're (perhaps willfully) ignoring the many reasons why photography is banned/strongly discouraged in some public places.

That's all - I just don't agree with you