News for hackers. Stuff that matters.
Re:I/O is no longer the bottleneck by throwaway71271 on Saturday November 26, 2022 @06:46AM (#33751885)
EBS costs crazy crazy amounts for reasonable iops

We pay 7k per month for RDS that can do barely 2k iops.. in the same time a machine at hetzner does 2 million iops for 250 euro per month (not to mention it also have 4x more codes and 5x more ram).

So, even though I/O is no longer the bottle neck physically, it still is a considerable issue and design challenge on the cloud.

Re:I/O is no longer the bottleneck by nix23 on Saturday November 26, 2022 @08:15AM (#33752360)
Well yes it's a total ripoff

I installed a DB-Server for a Customer around 2years ago, in a DC near him with 16 cores 48GB Ram and ~6TB -> 12 SSD, vDevs mirror with 2, and zRaid0 over the mirrored vDevs (kind of a Raid10), compression zstd (1GB could be compressed down to ~200MB so 5 times less reading/writing, and in theory ~30TB of pure DB-Data, 20TB realistic, remember never fill a zpool over 72%) and record-size 16kb. After 3 month the machine was paid (compared to the "cloud"-price) and the performance kind of 10-12 times higher.

Called the customer about a two month ago and he said the DB-Server is still to fast and maybe he wants another one who uses less power... ;)

Re:I/O is no longer the bottleneck by samb1729 on Saturday November 26, 2022 @09:51AM (#33753103)
> remember never fill a zpool over 72%

Could you please explain where this number comes from?

Re:I/O is no longer the bottleneck by ewwhite on Saturday November 26, 2022 @10:48AM (#33753556)
Yeah, that's a myth now. It's not current advice.
Re:I/O is no longer the bottleneck by nix23 on Saturday November 26, 2022 @05:12PM (#33757340)
72% is my rule of thumb for write heavy production stuff (my absolute limit would be 75%) but it depend on record-sizes, raidz-level, if you have mostly write or mostly read workloads, how big your files are, how many snapshots you have if you have a dedicated ZIL-Device and much more. For a Home-NAS (Movies etc) you can easy go up to 85%...if it's a "~WORM" workload maybe 90%...but resivering can then be a thing of days (weeks?), depends again on the raidz-level or mirror etc.

>Yeah, that's a myth now. It's not current advice.

It's not and you know it, keep it under 72% believe me if you want a performant zfs (especially if you delete files and have many snapshots...check the YT linked at the end)

>>Keep pool space under 80% utilization to maintain pool performance. Currently, pool performance can degrade when a pool is very full and file systems are updated frequently, such as on a busy mail server. Full pools might cause a performance penalty, but no other issues. If the primary workload is immutable files (write once, never remove), then you can keep a pool in the 95-96% utilization range. Keep in mind that even with mostly static content in the 95-96% range, write, read, and resilvering performance might suffer.

https://web.archive.org/web/20150905142644/http://www.solari...

And under no circumstances go over 90%:

https://openzfs.github.io/openzfs-docs/Performance%20and%20T...

>An introduction to the implementation of ZFS - Kirk McKusick

https://www.youtube.com/watch?v=TQe-nnJPNF8

Re:I/O is no longer the bottleneck by fomine3 on Saturday November 26, 2022 @10:19AM (#33753330)
Virtualized storage system increases latency (QD1 IOPS). Naively built non-parallel apps tend to rely on QD1 IOPS performance, so it runs very slowly on cloud platform, compared to dev machine with direct attached NVMe.
Re:I/O is no longer the bottleneck by nix23 on Saturday November 26, 2022 @07:15PM (#33758506)
>Naively built non-parallel apps tend to rely on QD1 IOPS performance

PostgreSQL is pretty much parallel but i know what you mean...the beehive ;)

Re:I/O is no longer the bottleneck by ilyt on Saturday November 26, 2022 @04:28PM (#33756943)
The cloud costs really are between "unreasonable" and "very unreasonable". The only time when it gets cheaper is if workload would be so spiky we coul'dve turned most of it off for 2/3 of the day but most of what we have is many smaller projects that don't even have enough traffic to even need scaling and the big ones, well, can't scale database size down off-peak...

Over last ~6 years we did "is it worth going to cloud" calculation few times and it was always ridiculously more expensive.

Re:I/O is no longer the bottleneck by LastTrain on Saturday November 26, 2022 @10:40AM (#33753501)
Agree! And at the end of the day, we are optimizing for cost. Although, the EBS portion of that 7k RDS bill is going to be tiny, right?
Re:I/O is no longer the bottleneck by throwaway71271 on Saturday November 26, 2022 @02:01PM (#33755530)
well its tiny if you keep it 2k, and make sure you dont touch disk much, god forbid someone makes a query that requires a temp table.. a query you wouldnt even notice in a baremetal machine brings the whole rds setup down, cant even read the writeahead log and cant replicate and etc.. its like watching a slow motion train wreck from 2 queries per second