News for hackers. Stuff that matters.
Re:The database servers powering Let's Encrypt by tyingq on Thursday January 21, 2021 @02:12PM (#25862779)
I was, long ago, an old-school Unix sysadmin. While I was technically aware of how powerful smallish servers have become, this article really crystallized that for me.

64 cores and 24 NVME drives in a 2U spot on a rack is just insane compared to what we used to have to do to get a beefy database server. And it's not some exotic thing, just a popular mainstream Dell SKU.

If you price it out on Dell's site, you get a retail price north of $200k. That is really what made it clear for me. That you could fit $200k+ worth of DIMMS, Drives, CPUS into a 2U spot :)

Re:The database servers powering Let's Encrypt by jaas on Thursday January 21, 2021 @06:16PM (#25865967)
Just so everyone here is aware re: pricing...

If you buy through a VAR and/or Dell reps you don't pay the price on the website. What you actually pay is typically significantly lower. I don't think anyone actually buys servers like these by just ordering from the website. We (Let's Encrypt) certainly don't.

These are expensive servers, crossing into six digits, but not $200k.

Re:The database servers powering Let's Encrypt by paxys on Thursday January 21, 2021 @09:54PM (#25867757)
Literally the only reason to display a retail price on the website is so sales reps can say "see we are giving you a massive XX% discount!"
Re:The database servers powering Let's Encrypt by midasuni on Friday January 22, 2021 @05:28AM (#25870337)
I wonder if such “price on application” behaviours is partially what drives services like AWS

I’m looking at buying some fortigate firewalls to do some natting, looks like 200Fs will be fine, but even the fortigate sales guys won’t give me a price, I have to go to resellers who also refuse to give prices, which adds friction. Cisco exactly the same.

When looking at options, price is at the forefront of my mind, but sales guys want me to choose their company, and even commit to the specific device and numbers, before I even see the price.

Almost walked away from the fortigate option until I found avfirewalls which gave me a ballpark idea of what I could afford to implement, and what trade offs. The benefits of the fortigate over a mikrotik were worth it at that price, but it was painful getting the price out and they nearly lost the sale as I assumed it would be 10 times more.

Re:The database servers powering Let's Encrypt by haltingproblem on Friday January 22, 2021 @10:14AM (#25872301)
You can back into the price from used gear.

Ebay has them for 3.2k and CDW new for 3.6.

Re:The database servers powering Let's Encrypt by mschuster91 on Friday January 22, 2021 @06:43AM (#25870792)
The same bullshit as US medical prices, it seems:

- Hospitals have an interest in massively inflated high sticker prices to make as much money as possible from those who pay in cash, e.g. foreigners, and to make the debts look better for collection agencies - basically, assuming the true cost of a procedure is 1000$, when the hospital bills 1000$ and that goes to collections who buy the debt for 10% of value the hospital gets 100$, and if the hospital bills 10k $ and the collections agency pays 10%, the hospital gets 1000$ or the actual cost

- Insurances have an interest in high sticker prices because they will negotiate with the hospital to pay true cost + some markup anyway - so with a higher sticker price, they can claim that their insurance saves the buyers a higher percentage

- Employers have an interest in high sticker prices because they can market themselves as employers who provide a better health insurance than the competition

The people losing out in this gamble are those who cannot afford insurance and have to declare medical bankruptcy.

And in the case of hardware or even some "contact sales for a quote" SaaS it's the same end result: the ones who lose out are small businesses (who can't achieve the sales amount to qualify for cheap-ish rates), and the big companies with dedicated account managers have a nice life.

Re:The database servers powering Let's Encrypt by chrisseaton on Friday January 22, 2021 @07:43AM (#25871193)
But don't they lose sales from people who never bother to call a sales rep because they think it's out of their price range?
Re:The database servers powering Let's Encrypt by deepstack on Friday January 22, 2021 @04:16AM (#25869893)
Sounds about right.
Re:The database servers powering Let's Encrypt by FlyingSnake on Friday January 22, 2021 @02:38AM (#25869382)
This is Anchoring, a cognitive bias where an individual depends too heavily on an initial piece of information offered to make subsequent judgments during decision making.
Re:The database servers powering Let's Encrypt by robk on Friday January 22, 2021 @02:02AM (#25869206)
There's a good thread on the sysadmin reddit where folks share the actual prices they're paying
Re:The database servers powering Let's Encrypt by FlyingSnake on Friday January 22, 2021 @02:38AM (#25869385)
Link for the curious?
Re:The database servers powering Let's Encrypt by skylanh on Friday January 22, 2021 @03:31AM (#25869642)
Re:The database servers powering Let's Encrypt by tyingq on Thursday January 21, 2021 @06:30PM (#25866145)
Yes, that's worth noting. The $200k is the list/retail, which nobody would pay for a purchase of this size.
Re:The database servers powering Let's Encrypt by walrus01 on Thursday January 21, 2021 @07:52PM (#25866856)
At this size/scale I wonder what the actual price would be for a comparable Supermicro system. If I had to make a wild guess, well under half the 200k previously quoted.
Re:The database servers powering Let's Encrypt by tyingq on Thursday January 21, 2021 @08:58PM (#25867305)
The NVME drives are what's making it so expensive. Dell retail is $6044 for each of the Intel P4610 6.4TB drives. That's $145k.
Re:The database servers powering Let's Encrypt by walrus01 on Thursday January 21, 2021 @09:00PM (#25867322)
after 15 seconds of googling, the P4610 U.2 format 6.4TB seems something more like a single unit street price of $2400 from non-Dell vendors. I'm mildly surprised it's that low considering that the U.2 format stuff, for serious servers, will always command a premium price.

Probably in the range of $2100 to $2200 per unit from a x86-64 component distributor in moderate quantities.

Re:The database servers powering Let's Encrypt by tyingq on Thursday January 21, 2021 @09:22PM (#25867489)
Just noticed this in the Dell cart...

"42% off list price: use code SERVER42"

Doesn't make the price reasonable exactly, but it's kind of funny.

Re:The database servers powering Let's Encrypt by knicknic on Thursday January 21, 2021 @11:54PM (#25868460)
You should try server43
Re:The database servers powering Let's Encrypt by k3liutZu on Friday January 22, 2021 @02:10AM (#25869255)
server100 for the win
Re:The database servers powering Let's Encrypt by bierjunge on Friday January 22, 2021 @06:00AM (#25870548)
server1000 and we can retire early
Re:The database servers powering Let's Encrypt by unreal6 on Friday January 22, 2021 @03:53PM (#25876229)
it seems like server-1000 is actually what one might be looking for
Re:The database servers powering Let's Encrypt by Dylan16807 on Friday January 22, 2021 @01:14PM (#25874345)
It looks like performance-wise I can compare it to a couple of 970 EVOs stuck together, and those are $160/TB retail. So from that point of view you're paying more than double.
Re:The database servers powering Let's Encrypt by giancarlostoro on Friday January 22, 2021 @10:14AM (#25872288)
Well, now I have to ask an old boss if he called a rep, he didnt spend quite 100k+ but we did get a twenty grand single server from dell about half a decade back or more, and now I'm curious, I think it was about 20 grand, I don't think NVMe's were mainstream at the time, or anywhere damn near affordable. I suspect he did buy it directly off the site, as he has other hardware we'd have gotten (mainly workstations).

In a way you could say the ones not calling a rep are subsidizing your cost for the server.

Re:The database servers powering Let's Encrypt by staticassertion on Friday January 22, 2021 @02:45AM (#25869411)
This goes for most things. We buy Dell laptops, and the price drops like 20-40% easy.
Re:The database servers powering Let's Encrypt by walrus01 on Thursday January 21, 2021 @06:19PM (#25866013)
One of the interesting things about threadripper and epyc systems with a lot of PCI-E 4.0 lanes on one motherboard, if your goal is not NVME disk I/O, but rather network throughput, it also provides the opportunity for a DIY approach to very high capacity multi-100GbE ethernet interface routers.

Such as a fairly low cost 4U threadripper box, running VyOS (ultimately based on a very normal debian Linux) with multiple Intel 100GbE NICs in PCI-E slots.

The Intel linux kernel (and FreeBSD) driver support for those is generally excellent and robust. Intel has had full time developers doing the Linux drivers for that series of cards since the earliest days of their 64-bit/66MHz PCI-X 10Gbps NICs about 17 years ago.

Also worth mentioning that FRR is now an official Linux Foundation supported project.

https://frrouting.org/

https://www.intel.com/content/www/us/en/products/docs/networ...

Re:The database servers powering Let's Encrypt by jart on Thursday January 21, 2021 @10:18PM (#25867910)
Altavista used to run their entire search engine on a single computer twenty years ago. Imagine what Mike Burrows could accomplish today with one of these 2U EPYC NVME 200gbps PCs. It'd probably trigger the singularity.
Re:The database servers powering Let's Encrypt by dan-robertson on Thursday January 21, 2021 @08:11PM (#25866975)
For a comparison, you could get a 64 processor, 256Gb ram Sun Starfire around 20 years ago[1]. Wikipedia claims these cost well over a million dollars ($1.5 million in 2021 dollars). This machine was enormous (bigger than a rack), would have had more non-uniform memory access to deal with, and the processors were clocked at something like 250-650MHz.

[1] https://en.wikipedia.org/wiki/Sun_Enterprise

Re:The database servers powering Let's Encrypt by tyingq on Thursday January 21, 2021 @11:57PM (#25868479)
That's a pretty good comparison. It weighed 2000lbs, and was 38 inches wide, or basically 2 full racks, which I guess you could call an 84U server. It was also 49 inches deep, versus a standard rack which is 36 inches deep.
Re:The database servers powering Let's Encrypt by angled on Friday January 22, 2021 @01:44AM (#25869086)
iirc, anton got a kernel compile down to about a minute with -j32 back in the day while at ozlabs?
Re:The database servers powering Let's Encrypt by vmception on Thursday January 21, 2021 @02:58PM (#25863409)
I have a motherboard from 2012 and I just put 2x 8TB NVMe SSDs on it, on a PCIe 2.0 x16 slot

Works great. The PCIe card itself has 2 more slots for SSDs

The GPU is on the 2.0 x8 slot because they don't really transfer that much data over the lanes.

I honestly didn't realize PCIe was up to 4.0 now, and I am pushing up against the limits of PCIe 2.0 but it still works! And I’m “only” at the limits, and its only a limit when I want faster than 3,000 megabytes per second, which is amazing.

Granted, this would have been considered a good enthusiast motherboard in 2012. Buying new but cheap is the mistake.

Re:The database servers powering Let's Encrypt by foota on Thursday January 21, 2021 @04:25PM (#25864474)
What drives did you get? I think you need PCI 4 to stress most SSDs these days?
Re:The database servers powering Let's Encrypt by vmception on Thursday January 21, 2021 @05:05PM (#25865053)
I have a hunch that the pcie card itself is most important as it is doing bifurcation.

So each drive acts like it has its own slower (but fast enough) pcie slot, and then the raid0 combines the bits back to double the performance.

Could be wrong but I get 2,900 megabytes per second transfers from RAM to disk and back.

And this is PCIe 2.0 x16

so maybe if you want , 3,000, 4,000 or 6,500 megabytes per second then I have nothing to brag about. I’m pretty amazed though and will be content for all my use cases.

Re:The database servers powering Let's Encrypt by infogulch on Thursday January 21, 2021 @07:13PM (#25866557)
STH recently did a video about this dense mass of storage & compute they call a server. iirc they specced it with their discounts to end up at a much more reasonable ~$90k.

https://youtu.be/9PLvt3WATMU

Re:The database servers powering Let's Encrypt by alberth on Friday January 22, 2021 @08:54AM (#25871635)
And if you don't want to buy Dell, you can get a SuperMicro server that has about 2x the specs for half the price ($102k).

https://www.siliconmechanics.com/system/a+-ultra-superserver...

2U

128-Core

307TB NVME

4 TB RAM

-------

= $102k

Re:The database servers powering Let's Encrypt by haltingproblem on Friday January 22, 2021 @10:18AM (#25872328)
We profiled Supermicro vs. Dell vs. HP vs. IBM servers a decade ago for trading applications.

Supermicro was cheaper by 20-30% had a quantitatively better power profile which basically meant it paid for itself and just rocked on every dimension. And their sales reps were able to answer every question we had including sending us a excel model of their power usage. The dell folks never got back to us on power usage.

I dunno why folks insist on Dell crapola.

Re:The database servers powering Let's Encrypt by segmondy on Thursday January 21, 2021 @10:15PM (#25867899)
I'm surprised that's what did it for me. I admined some sun enterprise servers and was blown away when I bought a pi 3 yrs and realized the pi was probably faster than it. That enterprise server would have 100-200 engineering students logged in and working at once. :-/. We're living in the future.
Re:The database servers powering Let's Encrypt by bloopernova on Friday January 22, 2021 @10:49AM (#25872661)
I like to play a game called "A current Raspberry Pi (or mobile phone) is equivalent to what year's global computing power?"

Also "fun" is "The solid state storage I can buy for $100 is equivalent to all the world's computing storage in what year?"

I'm super fun at parties!

Re:The database servers powering Let's Encrypt by spijdar on Friday January 22, 2021 @01:18AM (#25868954)
> the pi was probably faster than it.

If we're talking about 90s or early 2000s Sun, probably. Even then, though, these systems probably had substantially better I/O performance.

I have a SPARC laptop lying around from 1995 that gets a whopping 20 MB/s in disk read/write speeds across its dual half height SCSI drives. That still beats all but the best SD cards.

The full size systems with SAS and other options would get even better disk performance.

Re:The database servers powering Let's Encrypt by wpietri on Thursday January 21, 2021 @05:25PM (#25865315)
Totally. 2 TB of RAM! In one box!

I think the first servers I had in production had 8 MB RAM. No more, certainly. Soon we'll be at 1000x that. My dad's first "server" was 3 orders of magnitude smaller, with 8 KB of RAM (hand-wound wire core memory). In that time, the US population hasn't even doubled.

Re:The database servers powering Let's Encrypt by buzer on Thursday January 21, 2021 @07:28PM (#25866690)
https://yourdatafitsinram.net/

The listed servers are from few years ago so I guess there might be some bigger ones available these days.

Re:The database servers powering Let's Encrypt by jagger27 on Thursday January 21, 2021 @05:44PM (#25865583)
More like 1,000,000x
Re:The database servers powering Let's Encrypt by wpietri on Thursday January 21, 2021 @09:24PM (#25867509)
Oops! Yes, that's correct.
Re:The database servers powering Let's Encrypt by midasuni on Thursday January 21, 2021 @07:38PM (#25866747)
My phone in my pocket has 1,000 times 8MB of ram. A million of your first serves in one box. In about 30 years?
Re:The database servers powering Let's Encrypt by ggregoire on Thursday January 21, 2021 @08:02PM (#25866928)
We made so much progress in that field over the last 40 years... I sometimes get lost trying to imagine what will be the computing performance available in 100, 1000, 10000 years from now...
Re:The database servers powering Let's Encrypt by JohnJamesRambo on Friday January 22, 2021 @12:41AM (#25868707)
And I'm afraid it will still obey this trend.

https://danluu.com/input-lag/

Re:The database servers powering Let's Encrypt by kornbattery on Friday January 22, 2021 @04:18AM (#25869909)
That doesen't seem like some fundamental restriction, but rather a compromise we're always willing to make. 30ms is not noticeable, so we don't try to lower it and sacrifice something else.

Even the super slow ones like on kindle, are _choices_ that have been made in favor of something else. a second to turn a page on a book isn't unbearable

Re:The database servers powering Let's Encrypt by yakubin on Friday January 22, 2021 @08:38AM (#25871528)
30ms latency is acceptable. But that's the best we ever had.

High-end Android phones have 120+ ms latecy. That's easily noticeable and actually annoying (at least to me).

My personal pet peeve is the latency of input when i'm starting new applications in KDE. E.g. I'll start a new terminal with Super+Enter, followed by a Super+Right Arrow in order to tile it to the right. But the latency is big enough, that often it's not the terminal that ends up tiled, but the application that had focus earlier, e.g. a web browser. It's really annoying.

I also don't understand how still in 2020 when I move a window with the mouse, the window can't keep up with the mouse.

30ms input lag is what we should work towards. But that's not what we have today. Today is actually crap.

Re:The database servers powering Let's Encrypt by JohnJamesRambo on Friday January 22, 2021 @09:16AM (#25871822)
It’s so frustrating that our phones and computers can’t move at the speed our mind moves. I know it could if someone cared to try to make it a priority.
Re:The database servers powering Let's Encrypt by bloopernova on Friday January 22, 2021 @10:47AM (#25872627)
I think that if you had a top-to-bottom approach to a Linux distro designed entirely for latency of user interaction, you'd get something that would keep up with you.

I keep meaning to try out WindowMaker again as my Fedora window manager. I feel like it would be incredibly speedy on today's hardware.

Re:The database servers powering Let's Encrypt by gnarbarian on Thursday January 21, 2021 @08:18PM (#25867025)
Sticks and rocks most likely.
Re:The database servers powering Let's Encrypt by adolph on Thursday January 21, 2021 @11:44PM (#25868388)
Yep, quantum entanglement materials thru and thru
Re:The database servers powering Let's Encrypt by eppp on Thursday January 21, 2021 @08:51PM (#25867244)
Single threaded performance isn't keeping pace unfortunately.
Re:The database servers powering Let's Encrypt by SulfurHexaFluri on Thursday January 21, 2021 @11:38PM (#25868367)
That just means multithreaded will become more and more important. Languages like rust may gain some usage as we start to require safe multithreading.
Re:The database servers powering Let's Encrypt by eppp on Friday January 22, 2021 @08:25PM (#25878608)
That may be so, but that doesnt make it any less depressing. Some things cant be run parallel. I cant mail a letter until it is written.
Re:The database servers powering Let's Encrypt by theandrewbailey on Thursday January 21, 2021 @05:00PM (#25864999)
Dell-controlled VMWare is listed under "major sponsors and funders". I wonder if Let's Encrypt got a discount on that server. Good for them!
Re:The database servers powering Let's Encrypt by m463 on Friday January 22, 2021 @10:41AM (#25872562)
You can build a pretty nice home 32-core/256gb threadripper system for ~ $5k. (24 nvme drives not included)
Re:The database servers powering Let's Encrypt by amluto on Thursday January 21, 2021 @06:04PM (#25865830)
Dell makes it easier than it should be by having absurd markups on storage.