Remember that time a city in CA sent Cointelpro-esqe letters to the registered owner for vehicles their plate readers recorded going down a particular street?

There's nothing that prevents similar bad behavior from HOAs or cities/towns that have access to other data sets.

I don't want a HOA sending me brochures about recycling because my smart house reports that my ratio of trash to recyclables doesn't meet some arbitrary value. What goes on within the privacy my home is my business and mine alone to the fullest extent possible.

You can't abuse a data set you don't have. The value to society vs the potential harm of many datasets is so low they shouldn't exist at all.