News for hackers. Stuff that matters.
Re:Pingora, the proxy that connects Cloudflare to the Internet by nicoburns on Wednesday September 14, 2022 @10:59AM (#32838191)
> When crashes do occur an engineer needs to spend time to diagnose how it happened and what caused it. Since Pingora's inception we’ve served a few hundred trillion requests and have yet to crash due to our service code.

> In fact, Pingora crashes are so rare we usually find unrelated issues when we do encounter one. Recently we discovered a kernel bug soon after our service started crashing. We've also discovered hardware issues on a few machines, in the past ruling out rare memory bugs caused by our software even after significant debugging was nearly impossible.

That's quite the endorsement of Rust. A lot of people focus on the fact that Rust can't absolutely guarantee freedom from crashes and memory safety issues. Which I think misses the point that this kind of experience of running high traffic Rust services in production for months almost without a single issue is common in practice.

Re:Pingora, the proxy that connects Cloudflare to the Internet by drogus on Wednesday September 14, 2022 @02:04PM (#32840872)
I had a very similar experience. Much smaller scale, but the service was keeping internal state and clients were connecting with a WebSocket. It could handle up to a million clients on one server and it practically never crashed. While I was writing it I had only hobby-level experience with Rust and I was also mentoring a colleague, so he wrote a big chunk of code as a total Rust noob.
Re:Pingora, the proxy that connects Cloudflare to the Internet by WesolyKubeczek on Friday September 16, 2022 @09:22AM (#32865745)
I have a theory that people who are noobs in a certain technology, but are overall experienced, tend to produce very solid code, because they actively avoid doing clever things.
Re:Pingora, the proxy that connects Cloudflare to the Internet by 4m1rk on Sunday September 18, 2022 @01:10AM (#32884600)
Nice one. I believe we should avoid doing unnecessary clever things deliberately and actively.
Re:Pingora, the proxy that connects Cloudflare to the Internet by Aperocky on Wednesday September 14, 2022 @03:19PM (#32841980)
Curses! now I need to learn yet another language!
Re:Pingora, the proxy that connects Cloudflare to the Internet by bbarnett on Friday September 16, 2022 @07:36AM (#32864745)
Libcurses in rust, what a perfect idea!
Re:Pingora, the proxy that connects Cloudflare to the Internet by nicoburns on Wednesday September 14, 2022 @04:01PM (#32842592)
Luckily Rust is also fun :)
Re:Pingora, the proxy that connects Cloudflare to the Internet by victor106 on Wednesday September 14, 2022 @05:04PM (#32843341)
Is this using Async Rust?
Re:Pingora, the proxy that connects Cloudflare to the Internet by brink on Wednesday September 14, 2022 @11:07PM (#32846495)
Yes, if you want to handle that many clients, you'll need to use async. It's not too bad to learn.
Re:Pingora, the proxy that connects Cloudflare to the Internet by dcow on Thursday September 15, 2022 @12:45AM (#32847148)
Came here to post this quote too.

This is my experience (albeit at smaller scale) exactly. If this wasn't the case, Rust would be snake oil and I'd be the first to admit it. Until my most recent endeavor with a rust backend, among other things, I simply didn't know it was possible to not have to debug crashes. Seriously. The pleasure of maintaining rust software in production is so wildly different from anything I've ever experienced in the past that I gladly submit to the compiler over and over and over again. It's worth the investment 100 fold.

Re:Pingora, the proxy that connects Cloudflare to the Internet by brink on Wednesday September 14, 2022 @11:46AM (#32838824)
I had the same experience when I wrote a camera capture / motion detection / video logging service for a commercial smart refrigerator in Rust. The Swift component crashed at least twice a week, the Rust component ran for months and months without issue.
Re:Pingora, the proxy that connects Cloudflare to the Internet by jhgg on Wednesday September 14, 2022 @12:38PM (#32839649)
We had the same experience at work deploying rust services that serve many billions of requests a day as well.
Re:Pingora, the proxy that connects Cloudflare to the Internet by petr_tik on Wednesday September 14, 2022 @01:51PM (#32840714)
does your company have any public information about this? Blogs, job descriptions with numbers, twitter threads?
Re:Pingora, the proxy that connects Cloudflare to the Internet by remram on Wednesday September 14, 2022 @03:37PM (#32842246)
Not GP but I believe this is it: https://discord.com/category/engineering
Re:Pingora, the proxy that connects Cloudflare to the Internet by ellieh on Friday September 16, 2022 @05:50AM (#32864153)
Similar experience over here too, I've been running a Rust server in production for >1yr now and had absolutely no crashes. Resource usage has been nice and constant too. It's fantastic :)
Re:Pingora, the proxy that connects Cloudflare to the Internet by rkagerer on Wednesday September 14, 2022 @02:24PM (#32841161)
Which aspect(s) of Rust do you think are most responsible for this? (e.g. borrow checker, memory safety, culture that attracts devs who care about reliability, etc)
Re:Pingora, the proxy that connects Cloudflare to the Internet by gpm on Wednesday September 14, 2022 @04:07PM (#32842665)
Not the person you're asking, but the culture around data representation strikes me as the biggest factor:

1. Only making valid states possible, to the greatest extent reasonably possible.

2. Treating error as regular data, not an afterthought - with language features to make that not too painful.

3. Not returning placeholder values (i.e. if you get back a parsed value from a parse function, it means it parsed correctly, not that either there's an error somewhere else or it parsed correctly).

Language features, in particular "enums" (aka algebraic data types, aka tagged unions), make this approach possible. You couldn't do it in go, for instance, even if there was a cultural decision to.

Re:Pingora, the proxy that connects Cloudflare to the Internet by pornel on Wednesday September 14, 2022 @04:30PM (#32842937)
It is of course a combination of all these aspects.

A type system that can express thread safety (Send/Sync traits) is incredibly valuable when building multi-threaded systems.

Universal definition of what is safe, and standard traits and borrowing rules, make APIs more predictable. Just from function's signature you know a lot about its behavior, without having to look for gotchas in the manual.

Mandatory error handling prevents cutting corners. Unit testing is built-in.

Generics, good inlining, and Cargo help split code into libraries without a performance or usability hit, which helps make focused, well-tested components.

Most of these things aren't groundbreaking, but Rust being new had a luxury of picking current best practices and sensible defaults.

Re:Pingora, the proxy that connects Cloudflare to the Internet by otikik on Thursday September 15, 2022 @05:16AM (#32849179)
I think ultimately the big thing Rust brings to the table is that a lot of Rust's features is geared towards detecting problems as early as possible.

This means you get complexity, especially on the "initial phases".

* The initial learning curve is steeper than usual . You have x types of string instead of one or two, you have lifetimes, etc, etc. Fortunately you pay it once and then can use it many times (maybe with some revisits to the docs :P)

* The initial building of a feature might take a bit longer than in other less complex languages (the borrow checker gives you a particularly tricky error, etc). But what is happening here is that the language is picking up problems that other languages ... don't. They "defer to production".

Others are mentioning that cargo is another feature that helps. I think it does, but indirectly. What cargo does is improving the developer experience. Having all the packages at the tip of your fingers, without needing to jump over hoops like in other languages is just ... nice. Given the initial ramp up in complexity, and the initial harder-than-usual first write, anything that improves the UX goes towards alleviating that. I personally would put cargo on the same bucket as "helpful compiler messages" and "good docs".

Re:Pingora, the proxy that connects Cloudflare to the Internet by 0x457 on Thursday September 15, 2022 @07:57PM (#32860126)
Agree with you on most except for one point:

> You have x types of string instead of one or two

Funny how you said X, while there are really only 3 types of strings in standard library. You generally only encounter 2 of them: String and OsString. OsString, while annoying, but it's one of the things that makes rust safe. You only ever deal with 3rd type, CString, when you work with FFI, at which point you better already know how strings work on C.

Re:Pingora, the proxy that connects Cloudflare to the Internet by otikik on Friday September 16, 2022 @07:51AM (#32864854)
I said X because people include some and not others, and the conversation rapidly deviates (as shown on this thread). The point remains that there’s more ways to skin that particular cat in Rust than in other languages.
Re:Pingora, the proxy that connects Cloudflare to the Internet by nicoburns on Friday September 16, 2022 @05:53AM (#32864171)
There is also PathBuf, which would use a string in most languages. And sometimes one may need to work with Vec<u8> or Vec<Char>. And then slice/view variants of all the above.
Re:Pingora, the proxy that connects Cloudflare to the Internet by d3nj4l on Friday September 16, 2022 @09:06AM (#32865564)
Yeah, when Rust beginners (like me!) think "types of string" I think it's more likely to be &str (and its lifetime variants), String and Vec<u8>. I've gotten pretty far as a beginner by sticking to over-copying with just String, but eventually I think I'll have to figure out what works best for perf.
Re:Pingora, the proxy that connects Cloudflare to the Internet by 0x457 on Sunday September 18, 2022 @04:30PM (#32891216)
> There is also PathBuf, which would use a string in most languages.

I feel bad for them. I wanted to fix something in NeoVim's built-in lsp recently and lost all interest because of all the things you have to do to:

- Get parent directory for a path - Create that directory if it exists

Why was I annoyed by that? Well, different OSs use different path separators, in rust this is trivial because `Path` and `PathBuf` are target-specific.

That's not a type of string though, it's a path, the fact that it uses `OsString` underneath is irrelevant.

> Vec<u8> or Vec<Char>

There isn't really anything special to learn about it them, though? I doubt newbie encounter this while learning. Slices are the same thing, you just can't extend it, what is there to learn?

Re:Pingora, the proxy that connects Cloudflare to the Internet by manholio on Friday September 16, 2022 @07:19AM (#32864634)
> And then slice/view variants of all the above.

So all in all, about a dozen kinds of strings.

Re:Pingora, the proxy that connects Cloudflare to the Internet by nicoburns on Wednesday September 14, 2022 @05:58PM (#32843994)
A few things:

- I think memory safety is a baseline. You'll note that memory safe languages already tend to be much more reliable than non-memory-safe languages in general.

- Then you have the error handling. A lot of unreliability in my code in other languages comes from unhandled exceptions that only occur rarely. Rust generally puts all possible error conditions in the type signature of the function. Meaning it's actually feasible to handle every failure case.

- Speaking of unhandled exceptions, a lot of those in typed languages tend to be caused by null. Rust does not have null. Instead it has Option, and it is impossible to access the contents of an option without doing the equivalent of a null check. So that entire class of errors is gone.

- Both Result (used for error handling) and Option (used instead of null) are what Rust calls enums, and what are more generally called Sum Types. I think these are a huge deal. They allow you to safely represent data that may be one thing or another with very strict type checking. These are broadly very useful in API design, and in my experience lead to much more robust code than the class hierarchies you need in OOP languages or unions which lack the safety checks. (Aside: sum types would be quite simple to add to other languages. I have no idea why they haven't been added yet).

- Speaking of classes, inheritance is not supported. So that's a bunch of confusing code that just isn't possible to write. This can add a bit of boilerplate to Rust code, but it makes it more straightforward and less bug prone.

- You mentioned the borrow checker. That definitely helps. It's yet another tool that allows you to write APIs that cannot be misused. A great example would be Rust's Mutex type. It can prove at compile time that code does not hold on to references to the protected data beyond the duration that the lock is held.

- Speaking of Mutex. Rust's Send and Sync traits provide very good thread safety. You almost don't need to worry about thread safety at all in Rust. Most concurrency bugs are prevented by the compiler (you can still do things like cause data races).

- Newtypes allow you to check invariants once and then have the fact that they remain satisfied enforced by the type system.

- All type casts are explicit.

- Lots of other little things

One final thing that I think is often overlooked. Rust is strict, and all of these checks apply not only to the code your write, but to all of your dependencies. That means that Rust libraries tend to be much more reliable than libraries from other ecosystems. That probably is partly because of a culture of reliability. But it's also because the language itself makes it hard to write sloppy code. And that the code you are building on is likely to be reliable makes it both less effort and more worthwhile to make your own code reliable (including for library authors), leading to virtuous circle of reliable code.

Re:Pingora, the proxy that connects Cloudflare to the Internet by tomaszsobota on Friday September 16, 2022 @08:06AM (#32864977)
Thanks for taking the time to write it down, appreciate it.