The SafeBrowsing implementation does not send every URL you visit to Google. There's a local database of bad URLs, but stored in a way that can have false positives, to improve efficiency. Only if you get a match with the local database does it contact Google to see if it is a false positive or not.
I know that they are supposed not to misuse this information, but I would feel much better if they didn't receive it at all.