SMT is usually a throughput win, and usually a latency loss. I was playing with my Threadripper a while back and for C++ builds of large projects, HT results in about a 10% improvement in compilation speed. 10% is a big deal and you should take it. The downside is that games had noticeably lower framerates even with the rest of the CPU idle (at least the games I play are bounded by single-thread performance across maybe 2-4 cores). I kind of blame Windows's scheduler there, since it should be able to say "hey, a game is running, don't schedule anything on the same physical core that has a game thread running on it", but I don't think it does. It might schedule both game thraeds on the same physical core and then they contend with each other and run 40% slower each. Also be careful about memory -- 64GB wasn't enough for 64 concurrent clang runs. You need a little bit more (but of course 128GB is the next installable increment). (I can also see more threads aggravating other resource constraints; notably disk IOPS, but I didn't notice a problem there myself. It's also possible that SMT increases power use and so decreases turbo speeds, and that might have an impact. I didn't measure that when I was testing.)

For me, I keep SMT turned off. The latency is more important than the throughput for my workstation, but if you do full builds of C++ regularly, you might want it on. Use the 10% time you're getting back to switch to a build system that can cache things, though.