For private servers that are totally not connected to the internet in any way and don't need to be, run self-developed code on data which is mainly CPU-bound but also need a lot of disk IO for reading files and memory mapping data, I'm pretty certain it'll make a noticeable difference, for no security benefit.

but isn't it pretty easy to avoid the security update on those machines?

In the short term, maybe. But the OS update processes are incremental. If the user wants to upgrade their OS version for some reason (new feature, etc), they will get the performance hit bundled together.

For an average user that doesn't bother with security to the point that nothing on the server runs in sandboxes - Spectre and Meltdown are not of much concern. Running under a different user is not the same as running in a sandbox, certainly not on linux/bsd/windows. A user still has an enormous attack surface to play with. And due to the nature of multiuser software typically running on servers where all users live within the same trust boundaries any exploitable vulnerability is going to do as much damage, as if it was running under root, think Cloudbleed for example. You can't make things noticeably worse by turning off KPTI.

How do open ports factor in? And what if you’re internet connected but have JS disabled?

Open ports mean there is a service listening. A service listening means that there is code on your box interfacing with the internet. That means there is the possibility that an exploit in that code could lead to remote code execution. The meltdown vulnerability means that any remote code execution vulnerability can be escalated to the highest levels.

Exploiting meltdown means you can read all of kernel memory, that destroys any additional layers of protection you might fantasize you had. It means that in memory passwords and encryption keys can be copied. It means that IO between other processes can be spied on. It means that all of the juicy locations to exploit for "return oriented programming" in order to gain local root will be laid bare. And so on.

That's why it's called meltdown. Imagine someone who had hand crafted the equivalent of a modern OS with all of the best practices in place for an internet server. You have kernel address randomization. You have each service running as its own non-privileged user and also in a chroot jail. You have your filesystem permissions locked down tight. And so on. All of that is gone out the window with meltdown, it doesn't matter, because even code running at the lowest possible privilege level can gain access to the contents of any part of kernel memory.

As for "workstation" systems. If you are behind a firewall, you never run untrusted code, and you have JS disabled then potentially you are safe.

But on my home server, I don’t have layers protection and I don’t need them. If there is RCE that gets you into my RDP session, or my email, etc., you’ve got everything in the machine that’s worth having.

Or an internet connected or proxied web server with normal web server security precautions.