It also won’t have iLO, etc.

Even for a homelab these are useful features to have, I don’t want to go hooking up my monitor and keyboard every time I need to troubleshoot some boot issue or install a new OS. I can also get used DDR3 ECC memory for a hell of a lot cheaper than DDR4 right now.

Unfortunately the person who wrote this article is in the EU who has much slimmer picking in the second-hand market. I can buy a Dell R620 for $200-300, two 10-core Xeons for another $300 if it doesn’t come with enough cores, and 128GB of 16GB DDR RDIMMs for $200 - total price is under $1000USD, and that’s excluding any components that come with the server I may opt NOT to replace.

Useful iDRAC features like remote display and remote ISO install isn't in the base Dell iDRAC license. So I still need to hook up keyboard/monitor if it's more than a reboot.

Edit: iDRAC is the Dell LOM, iLO is HP

If you buy Supermicro motherboards, which come in regular form factors too, like for example X9SCM-F [0], or X11SSL-F [1], they come with IPMI (which is what the -F designates), which is exactly what iLO, DRAC, etc., is and in a normal form factor. You don't have to buy fancy jet-engine "server" boxes when you can get the same stuff in a mATX form factor. No license needed.

I have been using the X9SCM-F for the last 9 years, and enjoy being able to modify the BIOS remotely, boot remote ISO images, or troubleshoot any boot problems without physical presence.

[0] https://www.supermicro.com/products/motherboard/Xeon/C202_C2...

[1] https://www.supermicro.com/en/products/motherboard/X11SSL-F

Original parent comment was talking about installing a Ryzen consumer CPU, which is generally mutually exclusive with any board manufacturer (SMC, Dell, HP, or otherwise) that provides any sort of baseboard management controller.

Some of the SMC boards listed support i3 processors, but generally, if you want a BMC, you're stuck with enterprise-grade processors.

ASRock X470D4U is a server board, with HTML5-based KVM via BMC, which uses Ryzen consumer CPUs.

Wow, I wonder how many people knows about this? I was also expecting BMC to be EPYC, and Enterprise exclusive only.

It's often been sold out at Newegg, even with purchase limits. Price recently increased from $240 to $350.

No BMC license fee. Officially supports ECC UDIMM, Linux and PCIe bifurcation.

Mr Erbell, you made my day!

I listed two motherboards I personally researched in the past and knew about already. There are plenty of consumer Supermicro boards that have IPMI. Here's one I found just by browsing the first page of their workstation motherboard section. [0]

[0] https://www.supermicro.com/en/products/motherboard/X11SCA-F

None of yours are Ryzen, though.

Um, these are AMD EPYC (the server processor):

https://www.supermicro.com/en/products/motherboard/H12SSW-NT

All Supermicros for EPYC have IPMI.

This EPYC is $599: https://www.newegg.com/p/N82E16819113596?Item=N82E1681911359...

I'm not disputing, just seeing: this in the page from your previous comment:

Intel® 8th/9th Generation Core™ i9/Core™ i7/Core™i5/Core™i3/Pentium®/Celeron® series Processor, Intel® Xeon® E-2100 Processor, Intel® Xeon® E-2200 Processor. Single Socket LGA-1151 (Socket H4) supported, CPU TDP support Up to 95W TDP

Right, but the distinction we're making upthread is not Intel vs AMD, it's consumer CPU vs server CPU.

You can get out-of-band management with Xeon motherboards, or with Epyc motherboards, but you're likely not going to get a motherboard with out-of-band management with a Ryzen CPU or a Core i[3579].

But hilariously you can get IPMI with Atom motherboards

> they come with IPMI... which is exactly what iLO, DRAC, etc., is

iLO / iDRAC implement IPMI, but they are not the same thing. iLO and iDRAC provide web interfaces, remote iso mounting, granular boot control, remote vga, etc. There's a lot of useful features not available in a "standard" IPMI implementation.

You are confusing IPMI 1.0 with IPMI 2.0.

> Systems compliant with IPMI version 2.0 can also communicate via serial over LAN, whereby serial console output can be remotely viewed over the LAN. Systems implementing IPMI 2.0 typically also include KVM over IP, remote virtual media and out-of-band embedded web-server interface functionality, although strictly speaking, these lie outside of the scope of the IPMI interface standard. [0]

[0] https://en.wikipedia.org/wiki/Intelligent_Platform_Managemen...

I used to have X9SCM-F, and its implementation also had a web interface, remote ISO mounting and remote VGA.

Agreed; I also have X9SCM-F, and was able to do all that (which I already specified in my parent comment.)

IPMI does all of that and hundreds of more features. And it's licence-free.

https://www.supermicro.com/en/solutions/management-software/...

Again, IPMI is a standard which includes basically none of these features.

The BMC that vendors use implement IPMI and also usually provide these features.

I've got Supermicro -F servers and I've got PowerEdge servers with iDRAC Enterprise. Yes, the Supermicro servers have basic remote management features but the iDRAC Enterprise offers so much more.

If you're just running a couple servers in a home lab, yeah, it's probably not worth it but for a company with a bunch of servers the additional management features are absolutely worth it, IMO.

Redfish [0] is looking like it's gonna be pretty awesome too.

[0]: https://en.wikipedia.org/wiki/Redfish_(specification)

They are both different levels of bad. At least supermicro is free. The whole low level bios and BMC space is stuck in some 90s parallel universe. coreboot+openbmc can't come to the masses soon enough. It would be like what linux felt like after windows.

Sure, but the licenses to unlock them are easy to acquire.

Dell ties them to the service tag, which is quite easy to set via racadm and you can buy a license off eBay. Older Dell systems required a $30 part you can find on eBay. HP just uses a key they send you on paper.

Hell, many times these decoms still have the license applied to them. I have 2xR210 II’s, a R320, R520 and formerly had a HP ML10 and a Lenovo TD340. The Lenovo was the biggest pain to get VKVM because of the stupid hard to find dongle.

Don’t buy Lenovo servers for a homelab; BTW. Dell lets you override the ramp in fan speed when a non-Dell branded PCIe card is installed, newer Lenovo servers ramp them up to 100% and you can’t turn it off (I had to leave my TD340 on the initial BIOS revision with security issues because of this).

The 100% fan on non-whitelisted card is such a dick move. Is there a single legit reason for this ? Did someone's building burn down because they didn't buy the quadro gpu that Dell sells ?

It’s not even limited to GPU’s, a quad-port Intel NIC in my TD340 causes it to max fan speeds if I’m not running the A01 BIOS.

The argument is they don’t know the thermal requirements of the card because it’s not OEM-branded so they max the fans “to be safe”, but not letting you override it is just bullshit.

They do something similar on Chromebooks and 3rd party mini pcie WiFi cards. If you insert a non Lenovo branded card, it will fail to even enter the bios screen with a Dick Move error.

Also, same stupid justification,”this configuration hasn’t passed FCC EMI tests” when the cards a qualified directly, they are the FCC certification boundary.

I think lenovo has stopped doing that recently. However, we now have soldered wifi cards.

Back when I was in the HP VAR game iLO licenses where some of my highest money makers. MSRP for those licenses was like $250.00. I was buying them grey market from ESISO for like $35 a pop.

I bought ilo advanced license for my microserver for $5 at ebay. It worked just fine.

Probably from a keygen or duplicate use. But won't matter anyway. The ILO (or HP) will never know.

I have some principals with my homelab. Not doing grey market licensing is one of them. That's my personal preference, but just wanted to point out that iDRAC enterprise is additional.

> Hell, many times these decoms still have the license applied to them

I was indeed that lucky. Thank you <bank>.

Actually, remote display is possible with the base Dell iDrac Express license by enabling ssh console COM2 redirection. This is not enabled by default and not very obvious that it is even possible (probably because Dell wants to sell these iDrac licenses). For me the COM2 redirection is all that I need for a server in a datacenter without physical access. For example, if the server OS is not booting, I can still ssh to the idrac and change BIOS options or select alternative boot options in grub. For OS installs I use chroot. So really no need for an idrac license in my case.

Meh. I bought a $55 7" LCD screen and a $10 tenkeyless mini keyboard. Much cheaper and more flexible than iLO for how often I need to have non-network access.

I am running home servers for ~ 10 years, I never felt the need for iLO even when I moved the stuff in the basement. I have iLO on the servers at work, so I am not dismissing the utility, just the need at home.

It can happen once in awhile. I've been not-at-home the past few months and my mailserver fell over one day. I went to my apartment, turned it on, made sure mail was flowing, then went home. It had turned itself off again in the meantime. I drove out again, and turned it on, babysitting it for a few hours to see what might be happening. It's been up ever since, so, mystery. I would have liked a telephone/internet rebooter during all of this!

I’ve got 4 servers in my rack at home, I don’t like getting behind it to swap peripherals around. I plan on getting more eventually for a Ceph cluster too.

If you’ve got one or two, then, sure.

This is where I really appreciate front-side VGA ports.

I'm reinstalling Linux few times per year, so iLo comes very handy.

> It also won’t have iLO, etc.

You could install the free version of VMWares ESXi and use that as a sort of iLO. In most case that would be a more pleasant experience.

I'd really love to see a hypervisor that emulates IPMI/iLO/AMT/whichever flavor management hardware on machines that lack the capability. It should run a single VM only, and pass through all other hardware, but also cordon off eg. a single TCP port, and allow the user to power cycle the VM, along with remote control (could be done through VNC). Bonus points for exposing temperature and other sensors.

But when that hypervisor fails to boot you are SOL. I have vFlash cards in all my Dell servers with the OS install media and a recovery image so I can fix broken installs quickly. Oh, and that stupid PCIe training error from a NIC that went bad would have required I plug in my monitor as well.

It would be useful, but why not just run VMWare or Proxmox at that point?

At a potential performance penalty and significant extra attack surface (you now have to worry about your hypervisor’s attack surface and your actual OS’ attack surface, where as on bare metal you only have a single OS to worry about). Not to downplay the advantages of virtualisation, but if all you needed was a bare metal server then running a hypervisor adds one extra thing that can go wrong that you didn’t ask for.

Most (if not all) home lab users will be using virtualisation to spin up multiple VMs.

> you now have to worry about your hypervisor’s attack surface and your actual OS’ attack surface

But the same is true of iLO. Worse in that case, because the iLO won't receive security updates for as long and is less well scrutinized in general than common Linux distributions.

You can get a Tyan Tomcat EX or one of the ASRock Rack AM4 boards that have an Aspeed AST2500 chip to provide remote management. However, it might be difficult (and expensive) to get 4 32GB UDIMMs to work properly.

I have an x570 board with IPMI, it's not unheard of.

Waiting 2 months for Threadripper would probably be more sensible unless you need it RIGHT NOW though.

IPMI boards are increasingly available for AMD systems now. There is the Asrock X470D4U and X399D8A-2T and Supermicro also had a couple in the works I think.

Link!

That $1500 includes the rest of the computer, though. That adds up.

I almost went down this sort of road and went 3900x instead. I think the one non-replicated benefit of the older server route is cheap & large amounts of ECC ram, but you're also getting slower ddr3 ram which is why the pricing works out the way it does. I was in that same pricing ballpark with a new-build ryzen with a 2070 super, I think you could make it pretty much equivalent but maybe trading 64gb of fast ddr4 ram for the 128gb of ddr3. As long as you can "make by" with 24 cpu threads I think you'll be in a much better position at same-ish cost but with new / full warranty components but that's just my math :-).

correct, but you can get build a full system with the Ryzen 9 plus 128gb of ram etc for definitely less than $1700. Especially given the drop in ram and ssd's these days

128GB of ECC RAM itself is going to be ~800-1000$ as far as I can find some to fit in a Ryzen build. So it won't be trivial to reach the price point for a full build although Ryzen is clearly great value anyway.

True, but only if ECC is necessary. In this case where he is mainly using the server for experiments and to try software, regular ram will do absolutely fine.

Here is a 128gb kit for under $600 dollars. https://www.newegg.com/corsair-128gb-288-pin-ddr4-sdram/p/N8...

> only if ECC is necessary

If your workloads require 128GB of RAM then you probably want ECC. You'd be skimping in the wrong place.

This is kit of 8 sticks. Ryzen supports max of 4 dimms, so you need 32GB per stick.

> True, but only if ECC is necessary.

big "if" there really, and that makes it an apples-and-oranges comparison IMO.

I received a price quote on HPE ECC Server RAM this week from HPE that is $230 per 64GB stick. So make that $500 with shipping and tax for 128GB.

Consumer Ryzen requires unbuffered ECC RAM so most usual RAM options are not compatible.

SUPPORTS ECC RAM, but does not require it. You can use plain old RAM chips with it.

Grandparent's point, presumably, was that _buffered_ ECC RAM would not work. Which is definitely correct. I'm not sure if the sticks in question are buffered or not though.

Likely registered, not buffered - FBDIMMs fell out of favor some time ago. Technically still “buffered”, but only the control lines and it maintains electrical compatibility with UDIMMs.

Ah, thanks for that. I looked up the difference, I'll have to update my terminology.

Exactly, watch what happens to the price when you add 128GB of DDR4 memory.

I mean why isn't this the first comment reply.

The motherboard, RAM, SSDs, CPU, case, powersupply all have to be factored.

This is an apples to oranges comparison.

IMO it always makes more sense to build your own PC vs buy a used server.

It's going to be have better ventilation and be much quieter in a home setup.

Yes but build it with 128GB of ram and try and get a lights-out kvm on it.

Much better value can be had than OPs buy - just picked up a Dell R820 (quad 8core,3.2ghz v2 Xeons, hyperthreading is overrated yo) with 768GB of ram and dual 10G nics for $1500. Idles around 170W (about $5/mo where I live). The real magic: 8 full (16x 3.0) pcie slots.

The 3900x is a beast for the workstation though, and I have it running there with nothing but praise.

That's a great value!

I just checked eBay and it seems R820 with 128GB and 4x v2 Xeons goes for 1200-1300.

How did you manage to get the extra 640GB for 200-300 ?

PS 6 months ago I upgraded my 2x Xeon 2697v2 Dell Precision T7610 and paid $200 for 128GB DDR ECC (8x16) Didn't expect to see another price drop.

They come up often enough - someone will be selling a lot of 20+. They go quick but you usually have enough time to send a few messages before they sell out.

I bought an HP Proliant DL380 G7 with Dual Xeon Hex-Core X5660 at 2.8GHz, 76GB DDR3 RAM, and about 1TB of 15K HDD for $370.

On [H]ardForum, a kind user worked out for me a year ago the Handbrake performance comparison of a dual X5670 with some of the fastest consumer CPUs at that time: https://hardforum.com/threads/replacing-dual-x5670.1963388/#... turned out the 2700X is 1.95 times as fast in Handbrake as a single X5670.

I always forget that used servers are dirt cheap in the USA compared to other countries.

The other good thing about geting a Dell/HP rack server is you can have 16 drive bays (or more) easily.

The actual hardware is not bad, its just the perf per $ ratio is not great when you compare it to zen 2.

Kudos to AMD

Fair, but at $370 I think the ratio gets much better.

Not after you price the power. I have a Xeon system that is plenty powerful for my (C++) dev work, but I'm thinking of replacing it with an AMD build as it'll probably be cheaper over time. Then again, this is at Dutch energy costs, I'd imagine at (say) Texas cost it'd be different.

Not sure it's significantly better after factoring in power and auxiliary costs like cooling.

Where from?

Good point. If looking for server features on AM4 I highly recommend the ASRock Rack X470D4U motherboard for IPMI. Buying used HPE G8/9servers is still a great deal when you need redundant power and large amounts of RAM. Their 10g and 40g FlexibleLOM cards can be found on the cheap too. TechmikeNY is a great source of refurbished servers in the NYC area.

It seems like the author wasn’t targeting benchmarks exclusively, and comparing the price of a CPU to the price of a whole system isn’t fair. I spent about $2200 on a system that was targeting compute value- I have 4x E7-4890v2 for a total of 60 cores and can get >80000 on multicore geekbench. I also have 512GB of RAM, which I think is more than Ryzen can handle.

I paid $85 per CPU, so while a new Ryzen 9 does make sense for a lot of people with its fast single threaded speed and low power consumption, the old server gear I bought still wins in highly parallel tasks for less money.

Edit for proof of my geekbench score: https://browser.geekbench.com/v4/cpu/13853828 That was before I expanded the RAM, but the score didn’t change with the extra memory.

Thanks for this. How are the energy needs/costs?

I haven't measured power consumption with a meter, but each CPU has a TDP of 155 W, so potentially 620 W plus whatever is required to keep the motherboard, RAM, and disks alive. My ballpark guess is 750 W at full load and under 200 W at idle. It runs off a 1600 W power supply, but I suspect that is overkill. It's definitely less power-efficient than a new Ryzen CPU.

I have an almost identical setup hardware-wise (I opt'd for dual 2687 v2s [better single core performance]) and it only cost me $1000. It also supports 20 physicals disks with no extra hardware or configuration needed (aside from just plugging in the disks and adding 'em to ZFS).

RAM (DDR3) and storage capacity (cheap SAS drives) in these systems is really where the savings is... I don't think I'll ever own a "regular" desktop again.

Well that's used stuff, so Moore's laws ghost still in effect.

The biggest problem I have with a used server is the inability to incorporate a GPU into the architecture. But I guess that really depend on what you want to do with it, serving a website will absolutely not require a GPU.

Iirc dl380 gen 8 support 2x risers for gpu.

What kind of server does not support a GPU?

The form factor is (can be) an issue. I used a 4u case for my build a few years ago, and to fit my 1070 I had to bring out the dremel to remove some parts of the cross bars.

Something that was $1700 in 2013 is cheaper now? Shocker.

FYI the E5-2680 is like $300 on Amazon right now, so that's a better comparison.

The CPU was $1700. The entire setup is now less.

No worries, I am sure the Gen8 server makes for a good doorstop when it dies, have tested before.

With the ryzen 3950 with 16 cores coming soon for a bit more!

Notably more...

Epyc2 is what I'm fantasizing over lately.