News for hackers. Stuff that matters.
Re:How Discord supercharges network disks for extreme low la... by manv1 on Monday August 15, 2022 @08:33PM (#32477695)
To me, this sounds like a GCP failure. The provider is supposed to provide you with options that you need. If you're going to build it yourself why bother with GCP?

It would be a fun exercise to reimplement Discord in AWS...or with FoundationDB.

In any case before revving the hardware I'd want to know how ScyllaDB actually is supposed to perform. I mean, their marketing drivel says this right on the main page: "Provides near-millisecond average latency and predictably low-single-digit P99 response times."

So why are they fucking with disks if ScyllaDB is so good? I mean, back in the day optimizing drive performance was like step 1. Inside tracks are faster, and make sure you don't saturate your SAS drives/Fiber Channel links by mistake. It's fun to do, but you could always get better performance by getting the software to not do dumb stuff. Seriously.

Re:How Discord supercharges network disks for extreme low la... by nemothekid on Monday August 15, 2022 @10:03PM (#32478409)
>So why are they fucking with disks if ScyllaDB is so good?

The database layer isn't magic. The database can't give you low-single-digit P99 response times, if a single I/O request can stall for almost 2ms.

That said, I don't think AWS would fare any better here as the infrastructure issue is the same. Networked EBS drives on AWS are not going to be magically faster than networked PD drives on GCP. The bottleneck is the same, the length of the cable between the two hosts.

Re:How Discord supercharges network disks for extreme low la... by paulfurtado on Monday August 15, 2022 @11:55PM (#32479030)
At a huge price, EBS can finally get you near-local-nvme performance. If you use an io2 drive attached to a sufficiently sized r5b instance (and I think a few other instance types), you can achieve 260,000 IOPS and 7,500 MB/s throughput.

But up until the last year or two, you couldn't get anywhere near that with EBS and I'm sure as hardware advances, EBS will once again lag and you'll need to come up with similar solutions to remedy this.

Also, I guess AWS would fight them a little less here: the lack of live migrations at least means that a local failed disk is a failed disk and you can keep using the others.

Re:How Discord supercharges network disks for extreme low la... by briffle on Tuesday August 16, 2022 @11:20AM (#32483978)
google cloud also has a metric ton of IOPs and throughput on their networked persistent disks. But what this article is talking about is latency.
Re:How Discord supercharges network disks for extreme low la... by stingraycharles on Tuesday August 16, 2022 @05:25AM (#32480787)
What are the latency characteristics like for io2? You’re mentioning near-local-nvme performance, but describing throughput (IOPS can be deceptive as they are pipelined, and as such could still give you 2ms latency at times).
Re:How Discord supercharges network disks for extreme low la... by tuankiet65 on Tuesday August 16, 2022 @08:15AM (#32481792)
Apparently io2 Block Express (I'm not sure what's the difference from io2) is capable of "sub-millisecond average latency"[0].

[0]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/provisio...

Re:How Discord supercharges network disks for extreme low la... by zxcvbn4038 on Tuesday August 16, 2022 @02:09AM (#32479684)
I have this same conversation about AWS autoscaling all too frequently. It is a cost control mechanism, not a physics cheat. if you suddenly throw a tidal wave of traffic at a server then that traffic is going to queue and/or drop until there are more servers. If you saturate the network before the CPU (which is easy to do with nginx) or your event loop is too slow to accept the connections so they are dropped without being processed (easy to do I nodejs) then you might not scale at all.
Re:How Discord supercharges network disks for extreme low la... by kmod on Tuesday August 16, 2022 @07:23AM (#32481444)
I don't think it's the cable length: 0.5ms is 150km in fiber or about 100km in copper. Cable length is important in HFT where you are measuring fractions of micro seconds.

It's really quite amazing to me that HFT reduces RPC latency by about three orders of magnitude, I feel like there are lessons from there that are not being transferred to the tech world.

Re:How Discord supercharges network disks for extreme low la... by paulfurtado on Tuesday August 16, 2022 @12:01AM (#32479062)
Both GCP and AWS provide super fast and cheap, but fallible local storage. If running an HA database, the solution is to mitigate disk failures by clustering at the database level. I've never operated scylladb before, but it claims to support high-availability and various consistency levels so the normal way to deal with this problem is to use 3 servers with fast local storage and replace an entire server when the disk fails.
Re:How Discord supercharges network disks for extreme low la... by kodah on Monday August 15, 2022 @09:26PM (#32478161)
A public cloud exists to serve majority usecases, not to provide you with all the options. High speed, low latency network block io is probably not a common need at this point in time.
Re:How Discord supercharges network disks for extreme low la... by pojzon on Tuesday August 16, 2022 @01:26AM (#32479478)
Its very common issue. Decreasing transaction execution times for any type of request is the main goal for any data processing platforms.