I'm super cautious like you but one thing about the Mini and Mini 2 is they don't attract much attention. They're really small and pretty darn quiet.
I've had my Mini about a year and have yet to have any interaction with strangers who see it flying.
Random drones floating through neighborhoods is sketchy from a safety standpoint but also it tends to upset people, who often think that any drone they see is spying on them. Just a consideration.
Personally I like to fly around abandoned buildings and decrepit structures. Lots of interesting flying to be done and cool vids to be taken, and there's nobody around to complain :)