Yes I am worried, and not extremely happy about ownCloud. I was thinking about limiting access to it to my home IP but that limits the usability of sharing a link to some pictures etc with friends.

Don't get me wrong. The whole setup is both a burden & a liability as suddenly I am responsible for stuff that was done for me (security upgrades, proper configuration, monitoring, backups). That's the price I decided to pay for the ability to learn and control my own privacy. I may wake up to a hacked server - that's true. On the other hand I could wakeup to Google banning my account, then what?