emmericp

(Native German speaker here), it's a very rare use of the word "free" and usually only used in fixed terms like "Freibier", it wouldn't even work for other drinks, e.g., you can't say "Freisaft" or "freier Saft" for free juice, it has to be kostenlos there.

The other Wiktionary example of "freie Krankenversorgung" sounds wrong to be, but it seems to be used rarely in some more formal or legal contexts, no one would say it like this in a casual conversation. Google results also show a 4x difference between frei and kostenlos here in favor of kostenlos. But both are low since "Krankenversorgung" is already a very unusual word. I suspect many of those uses might be bad translations from English.

I'm surprised that the recent advances in applying typography to engineering problems [1, 2] are not published at SIGBOVIK but are apparently going to a more serious journal.

[1] https://www.researchgate.net/publication/390635826_Structura...

[2] https://www.youtube.com/watch?v=azDaPm13CT8

The time-travelling UB interpretation was popularized by this blog post about 10 years ago [1].

I'm not enough of a specification lawyer to say that this is definitely true, but the reasoning and example given there seems sound to me.

[1] https://devblogs.microsoft.com/oldnewthing/20140627-00/?p=63...

uIP (TCP/IP stack for tiny microcontrollers) is a another fun real-world example for these types of coroutines: https://github.com/adamdunkels/uip/blob/master/uip/lc-switch...

66% was drivers in 2019.

My aunt donated my grandpa‘s old slide rule there some time ago [0]. Crazy how far we came in just two generations.

Fun to see this site show up on HN :)

[0] https://sliderulemuseum.com/Circular/Curtis_L-M_FrequencyRes...

Yes, you should publish it. Don't bother cleaning it up if you don't feel like it. No one will judge you for the code quality.

Published terrible code is far better than unpublished code.

We wrote some experimental network drivers in high-level languages a few years ago and Java performed better than expected. Yeah, that's a very special and somewhat odd use case, but it was fun :)

We've got some graphs [0] comparing throughput and long-tail latency for the various Java GCs available in OpenJDK 12. Shenandoah's worst case pause time was ~45µs which is the same as disabling the GC (Epsilon GC) which is pretty impressive. Overall performance did suffer under Shenandoah a little bit back then, though. However, I've heard that this improved recently.

[0] https://github.com/ixy-languages/ixy-languages/blob/master/J...

Most "new tech" is engineering, not science.

If you want empiricism try academic papers instead of mailing lists ;) Some numbers: most critical bugs in the Linux kernel are due to memory safety: 40 out of 65 bugs allowing for code execution found in Linux in 2017 could have been prevented by using a memory-safe language [0]. The paper is about Go, but the same logic applies for Rust. Fun fact: 39 out of these 40 bugs were in drivers [1], so I think starting with drivers in Rust is a good idea.

[0] https://www.usenix.org/conference/osdi18/presentation/cutler (really good read!)

[1] https://arxiv.org/pdf/1909.06344.pdf (disclaimer: I'm a co-author of that paper)

The real root cause for all that flow director mess and core balancing is that there's a huge disconnect between how the hardware works and what the socket API offers by default.

The scaling model of the hardware is rather simple: hash over packet headers and assign a queue based on this. And each queue should be pinned to a core by pinning the interrupts, so you got easy flow-level scaling. That's called RSS. It's simple and effective. What it means is: the hardware decides which core handles which flow. I wonder why the article doesn't mention RSS at all?

Now the socket API works in a different way: your application decides which core handles which socket and hence which flow. So you get cache misses if you don't tak into account how the hardware is hashing your flows. That's bad. So you can do some work-arounds by using flow director to explicitly redirect flows to cores that handle things but that's just not really an elegant solution (and the flow director lookup tables are small-ish).

I didn't follow kernel development regarding this recently, but there should be some APIs to get a mapping from a connection tuple to the core it gets hashed to on RX (hash function should be standardized to Toeplitz IIRC, the exact details on which fields and how they are put into the function are somewhat hardware- and driver-specific but usually configurable). So you'd need to take this information into account when scheduling your connections to cores. If you do that you don't get any cache misses and don't need to rely on the limited capabilities of explicit per-flow steering.

Note that this problem will mostly go away once TAPS finally replaces BSD sockets :)

Lesson 1: Never ever reboot multiple Ceph nodes without checking if Ceph is happy between reboots. This failure happened early during boot and this could have been handled with no downtime if they checked the rebooted nodes before rebooting the next one.

Lesson 2: Avoid using RAID controllers except for the most simple "pass through" mode.

Lesson 3: XFS+Ceph never really worked out. BlueStore solved so many problems by just removing the XFS dependency for the actual data. Recommended reading: https://www.pdl.cmu.edu/PDL-FTP/Storage/ceph-exp-sosp19.pdf

ceph-volume finally fully removed dependency on file systems. Yeah the LVM-mess is sometimes annoying and early version of ceph-volume had many problems, but nowadays I wouldn't want ceph-disk back.

The NIC can hash on IPs and UDP ports just fine, nothing to worry about here.

Sure, there's the rare case of a connection moving to a different L3/4 flow identifier causing a cache miss, but that's not something that happens often enough to be a relevant bottleneck.

The hardware selection and test setup is questionable, yeah. But downclocking the CPU is pretty standard for many benchmarks like that. Would have been better to evaluate the effects of CPU frequency changes separately. But there's nothing wrong with downclocking in general if you want to evaluate effects of CPU bottlenecks when you don't have high-speed links available.

Isn't a large part of the performance gain from TSO offloading on transmit due to avoiding working with way too many skb's in the kernel? Shouldn't GSO for UDP with proper NIC support give you most of the performance gain?

I'm not sure about the implementation details of GSO and TSO in Linux. But I know a thing or two about the hardware: the Intel Niantic/82599 NICs support UDP transmit segmentation offloading, I've checked the datasheet real quick and it says "Note that current UDP segmentation offload is not supported by any standard OS."

Not sure if this is just an outdated comment in a 10 year old datasheet or if there's some performance to be gained by using this UDP segmentation offload for quick.

Edit: Should have clicked on the link in the article [1] before posting here, UDP segementation offloading is coming to lots of NIC drivers in 5.5.

Always interesting how the hardware often supports way more features than the drivers expose and you suddenly get new hardware features on a 10+ year old NIC model :) (Similar story with IPsec offloading on the Niantic/82599)

[1] https://kernelnewbies.org/Linux_5.5#Networking-1

> But enabling tcp timestamps in general case brings little benefit and wastes 12 bytes of each packet for basically no gain.

I disagree; TCP timestamps are awesome. Linux enables these by defaults.

Quick search gives me some measurements from 2012 [1] that indicate that TCP timestamps are enabled on 83% of the top 100k web hosts.

You can afford to waste 12 bytes; the bottleneck isn't these 12 bytes but how well you get congestion control to work. And congestion control relies on getting an accurate estimate of the round-trip time

[1] https://link.springer.com/chapter/10.1007/978-3-642-36516-4_... (paywall)

Edit: typo

Edit: Also, just because 83% of web hosts having it enabled does not imply that it is a good idea to do so in general. They could just all be running the linux defaults and these could be just wrong

There's only one level of TLBs in the IOMMU. And that's 64 entries.

Yeah, I think the dTLB is only 64 entries on Intel CPUs as well, but there's a second larger layer behind that, and an even larger third layer. IIRC it's a total of 4096 entries on recent Intel CPUs.

Interesting, did you use huge pages?

I think ~100k to 200k TSO "packets" per second should be doable with the IOMMU. But I guess it depends where the data is coming from. Could be one of the odd cases where copying data is faster than doing zero-copy, e.g., just copy everything into the same small set of small-ish buffers to keep the number of pages that need to be present in the IOMMU small?

A problem on the hardware side is that Intel's IOMMU TLB is tiny (64 entries), so using huge pages for all DMA-accessible memory is absolutely required to get a good performance out of it.

We've done some benchmarks here: https://www.net.in.tum.de/fileadmin/bibtex/publications/pape... (Figure 9 on page 10)

Only a very basic benchmark, working on more...

Related: I ran some benchmarks for all GCs in OpenJDK 12 some time ago: https://github.com/ixy-languages/ixy-languages/blob/master/J...

Epsilon tied on speed but lost to Shenandoah on latency because never freeing memory isn't ideal either, even if you never run out of memory.

Not a game developer, but I used to write UI addons for World of Warcraft. WoW allows you to customize your UI heavily with these Lua plugins ("addons") and Lua is garbage collected. It's a reasonable incremental GC so it shouldn't be too bad in theory.

But in practice it can be horrible. You end up writing all kinds of weird code just to avoid allocations in certain situations. And yeah, GC pauses due to having lots of addons is definitely very noticable for players.

Also leads to fun witch hunts on addons using "too much" memory, people consider a few megabytes a lot because they confuse high memory usage with high allocation rate... Our stuff started out as "lightweight" but it grew over the years. We are probably at over 5 MB of memory with Deadly Boss Mods (many WoW players can attest that it's certainly not considered lightweight nowadays, but I did try to keep it low back then). But I think we still do a reasonable job at avoiding allocations and recycling objects...

The point is: I had to spent a lot of time thinking about memory allocations and avoiding them in a system that promised me to handle all that stuff for me. Frame drops are very noticable. But there are languages with better GCs than Lua out there...

Somewhat related: I recently wrote about garbage collectors and latency in network drivers in high-level languages: https://github.com/ixy-languages/ixy-languages Discussed here: https://news.ycombinator.com/item?id=20945819

That's a scenario were short GC pauses matter even more as the hardware usually only buffers a few milliseconds worth of data at high speeds.

The Semaphore is only used during initialization, never in the critical path, see profiling results in the main repo

Yes, we do if we want to have an idiomatic interface for the application on top of the driver.

Pull requests proving otherwise are welcome

Yes, we could write most of the critical part in C and it would probably be faster. But then it wouldn't be a Swift driver.

Java reaches 52% of C speed in the benchmark game ("fastest measurement at the largest workload" data set, geometric mean), we reach 38%. Seems like our implementation is within a reasonable range for something that's usually not done in Java.

A full memory barrier is not required, but some languages only offer that. For example, go had the same problem. It's not a bottleneck because it goes to MMIO PCIe space which is super slow anyways (awaits a whole PCIe roundtrip).

And no, it obviously wasn't written by only one person but a team of 10.

No, we are not saying that we allocate for every packet. We say that we allocate 20 bytes on average per packet.

PCIe bandwidth also decreases with increasing packet size as there's a lot of overhead per packet. Memory isn't used, it's all handled in cache, hitting main memory is super slow.

The OCaml version is probably our most optimized implementation, it was scrutinized (and improved) by lots of people at the MirageOS retreat earlier this year.

Swift spends 76% of the time incrementing/decrementing reference counts; ARC is just very bad at pushing tens of millions of objects through it every second.

There's some more evlauation for Swift here: https://github.com/ixy-languages/ixy.swift/tree/master/perfo...

It's just a coincidence that JavaScript and Swift end up with almost the same performance; there is nothing similar between these two runtimes and implementations.

Real NIC drivers spend most of their time fiddling with bit fields. It's mostly about translating a hardware-agnostic version of a packet descriptor (mbuf, sk_buffs, ...) into a hardware-specific DMA descriptor.

If your driver copies memory you are doing something wrong.

Neither C# nor Java have C in the hot path; C# uses unsafe mode, Java sun.misc.Unsafe. JNI/C# native calls are either in the initialization or for an alternate implementation to compare them.

2-10% for an already fast user space driver is nothing.

State of the art for a lot of these use cases is still the kernel driver which is ~7 times slower. Sure, all that stuff is moving to XDP/eBPF/AF_XDP, but that is still ~20-30% slower than a user-space driver.

Also, these 2-10% only show up when underclocking the CPU while running the unrealistic benchmark of forwarding packets bidirectionally on only one core (trivial to parallelize).

In the end it's about 6-12 cycles spent more in the driver. That's not a lot if you have a non-trivial application on top of it.