In addition to wattage, my other concern for a home server was fan noise. If it's anywhere that I am, I really don't want to hear it. So I got these amazing Noctua fans and simply can't hear the server. (Updating language based on replies) A datacenter-like server probably isn't designed for that. So it wouldn't necessarily work out.

I have an 8 core i5. I bought Intel (18 months ago) because I didn't want to deal with any AMD incompatibility especially since I wanted to also run it as a gaming machine using VT-d. If I were doing it again, I'd definitely go with Ryzen.

I’m not sure what you’re getting at with the home server fans and not working out. I’ve been using a noctua on my Xeon home server without and issues, and reconfigured the fan sensors and control using ipmi. There’s no problem making a quiet Xeon home server.

I’ve got four servers and two switches in my home office, noise is around 45dBA and with the door closed it’s inaudible. Total power draw idle is around 340W (3.1A measure by my PDU @ 110V).

Sorry I wasn't clear. I meant something like a Rack mounted server designed for a datacenter. Would it become whisper quiet if you just replaced the fan? Is it that simple?

Replacing the fans on server systems is tricky, many “quiet” fans are such because they have a lower max RPM which will freak the hell out of your management controller. Also, there’s only so much optimization you can do to 40mm fans in a 1U chassis.

The fans in my 1U dell servers are fairly quiet when idle around 3600RPM. They make noise, you wouldn’t want to be sleeping or watching TV in the same room - but with the door to my office closed they can’t be heard in the hallway.

Ah I understand what they were getting at. If you're using a server motherboard in a regular 1-4u case this becomes a different issue entirely. I'm using a supermicro x9 with dual xeons in a upright enthoo pro case with it's own fans. You're right about the management controller freaking out because the IPMI threshold settings are expecting the supermicro 3 fan assembly instead of different case fans. Fortunately with some tweaking and ipmi tool you can effectively use fans that have a much larger or smaller range of acceptable RPMS without your management controller thinking the fans have gone down into lower non recoverable.

Right now my noctuas are running about 1000rpm and keeping the Xeons around 40c (under load this will increase with minimal db)

My solution was to take some spare Noctua low-noise adapters (basically inline resistors with fan connectors attached) and just drop the max fan speed. The fan controller can freak out all it wants, it can never go above about half rpms, and it still generates a noticeable breeze through the hotswap bays.

My CPU is a 7100 (ECC supported!) with a Noctua L9i so I never have problems there either. Power draw is a little high at about 70W with 8 3.5" drives spinning, but most of that is the HDDs (rule of thumb is 5W per drive) and the alternative would be spinning them down, which isn't ideal.

I think you meant an 8-core i7, there are no 8-core i5's.

Oops 6 core i5.

Worth noting:

> By default, this server is turned off. When I want to do some work, I turn the server on using wake-on-lan. I do this with all my equipment except for my router/firewall.

If it is only on when it is doing work, idle power doesn't matter _too_ much.

The Intel ark site says the TDP for that xeon is at 115W, and that's two of them. When you're running at full power (which could happen) you're looking at the CPUs alone drawing over 200W. Another pointer to high power consumption is the 750W PSU. All in all we're not looking at a particularly efficient machine (even if it's unfair comparing it to 2019 hardware).

The PSU's rating is for its potential to supply power, whilst draw components (CPUs, disk, etc.) are rated by use.

The actual efficiency is defined by capacity, load, and miscellaneous other factors, but should be in the neighbourhood of 20% of delivered power.

If the system idles at ~100 watts, the PSU might add another 20-30 to the mix, not 750.

https://www.howtogeek.com/129667/what-exactly-does-the-watta...

Agree with all of that, but I think OP was referring to the fact that higher-rated PSUs will use more power than a lower rated PSU when idle.

Running a PSU at ~50% of its rated max is generally recommendable. At 115W * 2, plus other draws (disk, mobos, blinkenlights), that's around 300W.

Again: peak draw, when on.

I'm not arguing that this system is particularly efficient, only that you don't want to add 750W for the PSU to the draw.

I've not specced out low-power systems myself. I doubt you could get 40 hyperthreads running way below this, though there are definitely some low-power systems which might have a total budget below 50W. Reddit's HomeLabPorn may have some more useful guidance: https://old.reddit.com/r/HomeLabPorn/

(Not my area of expertise, I've never really stayed current in HW. Though I'm aware power/thermal budget has been a major focus, both mobile and server, for the past decade or so.)

The 24 cores/48 threads EPYCs that come with 128mb L3 cache have a TDP that's between 155W and 180W. If you want to go with a lower core count (because you have a more modern architecture), you could even go with the 16c/32t EPYC 7282 (64MB L3) that sits at 120W. All in all, you're looking at almost half the power draw (AMD measures it differently than Intel, so aboutish the same) for the same performance.

I specifically mention the L3 cache size because, while not being substantial, a large cache can get you 10 to 20% improved performance due less CPU stalling from cache misses. For comparison, the Xeon in question has 25MB L3, so we'd be looking at 50MB cache, split in two dies (so it doesn't quite work as a whole block of 50mb cache).

greatjack613 also mentioned the fact that an AMD Zen 3900x matches the multicore performance (and has better single core perf) for a fraction of the TDP, see https://news.ycombinator.com/item?id=20687932.

All in all that was the general idea: yes, you can get a used server for a good price, but we shouldn't forget the efficiency aspect of it, compared to new hardware.

Good points, and again, not something I keep current in, generally, as the field changes so quickly, particularly relative to my own replacement cycles.

I turn this server off when I'm done experimenting. I turn it on remotely using wake-on-lan.

What exactly do you use it for if it isn on 24/7?

I bought a used intel Nuc a while back with 15w idle. And it runs 24/7 and does everything I need from downloading to nextcloud to plex, my website, my emails etc...

I really wonder what you do with this kind of power.

To contextualize this, ~100W will probably cost between $10-$15/mo to run 24/7.

EDIT: Assuming a US household paying between $0.10-$0.20 per kwh.

Where I am it costs 37c per kWh. It's no wonder almost everyone has solar.

You know, that raises a question - how expensive does power get, anyway? Last time I posted my home lab setup (couple of old R610s, Cisco small business switch, ASA 5510), I got a lot of pushback on how much power that consumes and how expensive it is to run. I did the math, and I was looking at maybe $20-$30/month depending on load. Which is not really a lot of money considering how much use I get out of it.

I'm definitely getting cheaper power than a lot of people at $.08/kWh, but it looks like the US average is only about $.12 - are there places where a couple hundred watts is going to be a significant financial burden on the average IT worker?

To me it's less a financial burden and more the discomfort of burning electricity just to run a few Docker containers and some databases for personal projects.

I expect that once, next summer, we get the house kitted with solar and batteries, I will feel a little more okay about it.

Some people like me will justify it due to being a huge hobby for them. It takes the place of what other people would spend going out often, traveling, streaming, etc...

I've got a rack that pulls ~600W 24/7 but that's across 4 machines, 3 UPS's, >30 HDD's yielding >20TB across multiple zpools, >256GB RAM. And I'm actually using most of that capacity, not just idling away.

I do hope to upgrade soon to power-sipping platforms like you mention but currently I'm still on R710/R810 stuff (Westmere Xeon's).

That is awesome. Do you have something written up about your setup online?

I have to ask, using it to do what?

I wish I had something written up to share. But it's honestly a mix of things I love to play with. Several full blockchain nodes (especially uploading to light clients), bittorrent client (seeding tons), plex, home assistant, boinc, tor bridge, grafana, web and email servers, zoneminder CCTV, mandelbulber (clustered rendering of 3d fractals), four different Minecraft servers (some modded), ADS-B plane tracking, weather station and radiation monitoring, and a bunch more.

It's really just one big toy for me. I'm using Proxmox for high availability of over 30 VM's (still want to play with containers soon).

EDIT: OPNSense firewall says I uploaded over 15TB in the last month. Fortunately Google fiber doesn't care!

That's awesome!

Linux ISO plex server.

Probably an error in measuring the power consumption, there is no way to go to 15W, double that or more.

It's believable (if only obtained with a lot of BIOS tweaking), I run an older Dell 1u server as my router. It's got an E3-1220 V2 (2012 era, 69 watt TDP), one 2.5" SSD, 8gb of RAM and after a bit of tweaking it sits at around 30 watts.

It wasn't a lot of BIOS tweaking, but yes, some undervolting was required. ;)

I also hdparm'd it such that my spinning disks park pretty aggressively.

Did you do anything particular to get 15W? I have recently measured my AMD AM1 system, which sits at 40W with 8GiB RAM, 2 HDDs and a couple mostly idle VMs on it.

I'd check the BIOS to see if there are any power saving options, and/or unclocking/undervolting the CPU.

2.5" HDDs and SSDs use about the same amount of power, but 3.5" HDDs use 2-3x more. So you could potentially save 12 watts if you're using 3.5" drives.

Make sure your PSU is a high efficiency one, and isn't too low, or or too high wattage. From what I've read the maximum efficiency is in the 40-80% range.

Maximum efficiency is at 50%

I do undervolt it and I aggressively spin down the drives. (My concern is mostly storage and redundancy, not performance.)

Or it's a ton of compute in a single system for just 96W. Average US rates are roughly $1/yr per continuous watt.

>idles under 15W

Damn...meanwhile my fairly modern Asus laptop idles at 27W.

(Probably the GPU...think I forced it to dedicated only)

I have a UniFi USG, UniFi 150W switch (using maybe 30w), a freeNAS with a i3-8100, a mATX super micro board and 6 4TB WD Red drives. My backup power supply says maybe 120w.

My USG runs very hot, do you experience it?

All my unifi stuff runs hot. The ERLite 3 and switches, etc. I think they count on using the cases as heatsinks.

Same here. Those things get toasty. I accidentally placed a RPi3 (passively cooled but with a metal case) on top of my USG once and it got hot enough not just to throttle it, but also to cause HDMI artifacting.

Not really. I have a NanoHD and my father in law has one too. His runs really hot and mine is cool to the touch. Mine has more traffic and was made before his NanoHD.

I also have a NanoHD and its acceptable but USG (non-pro) runs so hot that I fear leaving it in a non airconditoned cupboard that my internet exit resides

Yup. My USG and US-8-60W both run hot enough to not want my hand on for very long (the power supplies are pretty warm too).

That costs you maybe $.5 dollars a day? Maybe $1 if you are unlucky. This seems fairly reasonable?

I've done a measurement with a power meter and it's even 110W. (it includes ilo).

would newer hardware idle at even lower wattage than 15W? Or is that about the best one could expect?

Newer, maybe slightly but not hugely. (It's all still kinda warmed-over Skylake.) You could get cheaper with lower-TDP chips, but there are definitely diminishing returns on the floor.

For a full desktop class chip, I think that's about as low as it can go. You could go lower with something like a NUC, but 15 watts is really good. 15 watts idle would cost something like $1.50 to $2 to run 24/7.