Even asking "how was your weekend?" -- its implied that you just say good rather than sharing details of what you actually did -- they don't really care.
Building a cohesive team involves sharing interactions like this. If you truly value your privacy, I suggest you come up with an alternative thing to discuss: sports, weather, funny stories, etc.
I've certainly had weekends where I very much did not want to share what i did, so I deflected in a way that kept the conversation going.
If HR is asking about where you were at 3:15pm on Saturday, and who you were with, and whether you were using work resources at that time, I would start to worry. But a very large and generic, "so what did you do this weekend" - where you are in the driver's seat, that's not the same thing.
When I look back at previous jobs, my strongest regrets are not building stronger interpersonal relationships with my coworkers. I've since rectified that.
Think of it this way, if you must: coworker relationships are owned by the company - personal relationships are owned by the employees. When you become friends with a coworker, you acquire an asset that will go with you when you leave the company. Much like a new skill, this asset will allow you to get higher pay somewhere else.
One of my favorite staff/principal engineers always begins Monday's by asking people what they did on the weekend. He always talks about the things he did - farmers market, local parade, etc.
It's shocking to many of the junior engineers. They generally do "nothing".