Even if the site is HTTPS protected, a surveillance actor on the net would still be able to read the entire site, maybe to determine if the site has content worthy of tracking those who access it.
And, surveillance would still reveal that your IP address is accessing the site, and thus triggering something.
What HTTPS would protect is the specific URL path you are going after on the site, because that's in the HTTP GET which is part of the encrypted data traffic.
I guess you could say that maybe the site has some pages that are more sensitive than others, and revealing the exact URL paths you are accessing might set off a surveillance trigger that would otherwise not be noticed. But, the site in question is probably not like that.
If visits alone are problematic, it seems to me that the only possible solution is Tor.
That in itself may be considered a breach of privacy, as it exposes your users to passive capture and profiling.
Also, accessing the content you are hosting might be considered legal in some countries but illegal in others, regardless of if it is public or not.