News for hackers. Stuff that matters.
Re:We rewrote a high-performance database in Rust by kaladin_1 on Tuesday October 18, 2022 @05:08PM (#33253990)
I have no problem with people rewriting their projects in whatever language they see fit.

What stood out for me in the article is him saying that it's difficult getting developers with experience in both Python and C++.

So, I wonder, if his in-house devs could pick up Rust that they previously couldn't write, why does he think he can not hire a good programmer and charge him to learn the stack the company uses. Why must they employ someone that already writes Python or C++.

Is Rust such a straight-forward language that people new to the language can write a very performant programme

Re:We rewrote a high-performance database in Rust by pornel on Tuesday October 18, 2022 @05:21PM (#33254175)
Despite Rust's steep learning curve, it's also paradoxically easy to add novice Rust programmers to a project.

This is because inexperienced Rust programmers are relatively harmless. Noob mistakes won't compile, rather than running into dangerous gotchas. You can tell noobs not to use `unsafe` (and there are ways to enforce that), and mostly they'll just write inefficient or non-idiomatic code, but the code will be free from data races and memory corruption.

The strictness of the Rust compiler is quite the opposite of something like the C++ Core Guidelines where the majority of the rules aren't enforced by the compiler, and have to be in the programmers' head first.

Noobs make lifetime errors and fight the Rust compiler, but imagine working with a compiler that doesn't tell you when you have lifetime errors.

Re:We rewrote a high-performance database in Rust by krona on Tuesday October 18, 2022 @05:32PM (#33254300)
Memory sanitizers, address sanitizers, leak sanitizers, threading sanitizers, undefined behaviour sanitizers. The visual studio core guidelines checker. The clang-tidy core guideline checker. I could go on but my point is, the landscape does not really look like how you've painted it.
Re:We rewrote a high-performance database in Rust by pornel on Tuesday October 18, 2022 @05:40PM (#33254395)
I know about these, but there is a marked difference between Rust and these tools.

Static analysis tools have much harder job with analyzing C++ (aliasing and escape analysis is way harder, and static analysis of thread-safety is basically impossible due to lack of thread-safety info in the type system). The results are a trade-off between sparse or having false positives.

The sanitizers only catch issues they can observe at run time, and that relies on having sufficient test and fuzz coverage. Some data races are incredibly hard to reproduce, and catching vulnerabilities that require combination of multiple bugs or unusual specially crafted inputs is even harder.

OTOH Rust proves absence of these issues by construction, at compile time.

Re:We rewrote a high-performance database in Rust by krona on Tuesday October 18, 2022 @06:02PM (#33254640)
The results are a trade-off between sparse or having false positives.

Rust just takes the other side of the trade-off, and will reject valid programs. Hence why the unsafe keyword exists, and why tools like Miri (https://github.com/rust-lang/miri) exist specifically for rust.

Re:We rewrote a high-performance database in Rust by timeon on Tuesday October 18, 2022 @06:13PM (#33254757)
> Rust just takes the other side of the trade-off, and will reject valid programs.

Are we still talking about ease of add novice Rust programmers to a project?

Re:We rewrote a high-performance database in Rust by tialaramex on Tuesday October 18, 2022 @07:37PM (#33255557)
Sure, that's the consequence of Rice's theorem. "Is this program correct?" is formally Undecidable, to dodge that we split the programs three ways: Correct, Not Correct, shrug emoji.

It's obvious that Correct programs compile, Not Correct programs result in a compiler error, but what do we do with shrug emoji ? Rust says those go in the "Not Correct" pile and get a compiler error. C++ says - really, I'm not making this up - that they go in the Correct pile.

There's an immediate short term consequence, but also, after decades, a long term consequence that's arguably worse. Short term, C++ programmers can't know if their non-trivial program is Correct. It might be complete horse shit, the compiler won't necessarily tell them.

Long term, C++ grows worse and worse because there is no reason to shrink that shrug emoji category. All the programs in that category act as though they're fine, so there's no pressure whatsoever to improve the language standard, the compilers, etc.

Re:We rewrote a high-performance database in Rust by anyfoo on Tuesday October 18, 2022 @07:47PM (#33255624)
Over the decades, C started to reject more and more "valid" programs, and you have to use explicit casts to go from one pointer type to the other, or from an integer to a pointer. For obviously good reason.
Re:We rewrote a high-performance database in Rust by saghm on Tuesday October 18, 2022 @08:45PM (#33256031)
> Rust just takes the other side of the trade-off, and will reject valid programs

This has always been such a weird claim to me because it's not clear to me what's meant by "valid". My instinct is that it would be hard to define valid in a way without making most languages either accept some invalid programs or reject some valid ones (the exception being defining "valid" as "anything that language X accepts", but that wouldn't really say anything interesting about Rust). Sure, you could draw the line so that Rust is the only one that rejects valid programs, but is it worse to be the only language that rejects some valid programs than to be a language that accepts invalid ones? The alternative is that all languages accept some valid programs, at which point there's nothing specific about Rust that's worse until you start specifying which valid programs each language rejects. If there's a way to define "valid" so that Rust is the only one that rejects valid programs but other languages accept all valid and reject all invalid programs, I think it's non-obvious enough that it should be explicitly stated.

Re:We rewrote a high-performance database in Rust by Omin on Friday October 21, 2022 @12:30PM (#33289885)
That definition of what is valid or not depends on the type of checker. When you're talking about Rust's borrow checker, then you can take any property that it checks for, such as "there are no aliasing mutable references" or "any variable that is read from has been initialized" and build an example for it that would be correct, but is rejected by the borrow checker.

Here is an example where a variable is definitely initialized, but it will be rejected.

  fn main() {
      let foo;
      if true {
          foo = 1;
      }
    
      println!("{}", foo);
  }
Re:We rewrote a high-performance database in Rust by djwatson24 on Tuesday October 18, 2022 @06:17PM (#33254792)
> and that relies on having sufficient test and fuzz coverage

At the faang I worked at, some small portion of servers ran the sanitizers in prod, so you’re not reliant on test coverage nearly so much for catching rare issues.

Re:We rewrote a high-performance database in Rust by abc_lisper on Tuesday October 18, 2022 @05:36PM (#33254345)
Yeah, but Rust guarantees sanity by design. Sanitizers are a patch, and hence not comprehensive.
Re:We rewrote a high-performance database in Rust by lijogdfljk on Tuesday October 18, 2022 @05:37PM (#33254359)
They painted it like reality, though, no?

You seem to paint the landscape as full of tools and imply that they're used. Either they're insufficient or they're often under utilized, simply due to the number of bugs we see. No?

Re:We rewrote a high-performance database in Rust by atoav on Tuesday October 18, 2022 @05:53PM (#33254547)
All of these tools and we still have exploitable buffer overflows in 2022. So either the tools are not working, people are not using them or they are using them but can simply ignore critical warnings.

Your milage may vary, but I think Rust offers a well considered step into the right direction. Stupid and dangerous code of the kind every developer will produce once in a while just won't compile in Rust. You cannot forget to run a check, you can't hide behind not knowing a tool. You can't ignore the warning of you want a running program.

That is not nothing.

Re:We rewrote a high-performance database in Rust by nicoburns on Tuesday October 18, 2022 @07:13PM (#33255349)
Would you trust someone new to C++ to use these properly? The great thing about Rust is that you get all this and more (Rust's static checks are a lot more watertight) right out of the box without any configuration.
Re:We rewrote a high-performance database in Rust by maccard on Tuesday October 18, 2022 @07:29PM (#33255496)
Assuming you have a CI suite dedicated to haning all of the above, sure. Meanwhile some of us are staring down the barrel of 30-60 minute compile times for a single configuration. Multiplying that out to compile with ubsan, asan, and running the static analysis tools over it would take probably 5-6x longer, _and_ we likely need to do it all twice to ensure were checking our "shipping" code paths.

It's not feasible to have a developer compile for 3 different platforms in two configurations with and without sanitizers, and core guidelines checkers for every change, and these tools take so long it's a huge cost.

Re:We rewrote a high-performance database in Rust by Arcuru on Tuesday October 18, 2022 @07:29PM (#33255492)
I think it's great that Rust is safe in that way, where you can have novice programmers write code and you can be pretty sure it won't break anything else.

I recently spent some time fixing performance issues in some novice Rust code, and while the code was pretty clearly written by someone new to systems programming it still all worked fine - https://jackson.dev/post/rust-coreutils-dd/

Re:We rewrote a high-performance database in Rust by ekidd on Tuesday October 18, 2022 @05:31PM (#33254297)
I've written quite a bit of production code in C++, Python and Rust, and currently work on a hybrid Rust/Python system. Here's my experience:

- C++ is an unusually large language. And it has many historic footguns, requiring a higher level of vigilance and code review. If I were starting a brand new project today, I wouldn't try to build a team of C++ programmers.

- Untyped Python becomes more difficult to refactor and maintain once you reach 50k to 80k lines on a group project. Typed Python, however, scales nicely beyond this size.

- Rust is a "medium-sized" language. It requires developers to learn more than Go or Python does, but less than C++. And Rust has far fewer traps for the unwary and the reckless than C++. Rust's tooling is also very good in many areas.

- It's tempting to split a project into a fast "core" language, and high-level "glue" language. There are real advantages to this. (Which is why I've done it on one recent project!) But this also comes with costs: everyone needs to be fairly good at two languages, and switch back and forth. And you pay a tax at the boundary.

If I were building a brand new database (and a team to maintain it), I'd actually be strongly tempted to use Rust exclusively. But this is partly because databases rarely have a "business logic" layer that changes constantly, so there's less need for a high level scripting language.

But with a different team or different constraints, C++ could also be the right choice.

Re:We rewrote a high-performance database in Rust by hutzlibu on Tuesday October 18, 2022 @05:45PM (#33254466)
"But this also comes with costs: everyone needs to be fairly good at two languages, and switch back and forth."

Why does everyone needs to be good at both languages? You can seperate and have the core people writing efficient low level code - and you have higher level scripting/gluing code.

Re:We rewrote a high-performance database in Rust by pornel on Tuesday October 18, 2022 @05:55PM (#33254570)
You will have Conway's law in your codebase. Coordination between teams is hard, so teams will prefer to implement features entirely in their language, even where that is technically suboptimal.

You will get hot loops in Python, because a Rust programmer wasn't around, and Rust programmers implementing whole complex business logic in Rust behind a single `do_it()` Python call.

Re:We rewrote a high-performance database in Rust by hutzlibu on Tuesday October 18, 2022 @06:12PM (#33254744)
Communication and coordination is surely hard and things like that surely happen, but this is why project management exist.

If it is doing things right, then the rust people don't do complex buisness logic, because it is not assigned to them and they would not even have the details.

And if the python people were too eager and have core stuff implemented and it is affecting performance, than you can always reimplement it low level.

It all depends on the project of course, of what would be the best mix.

Re:We rewrote a high-performance database in Rust by lupire on Tuesday October 18, 2022 @05:47PM (#33254488)
A brand new project doesn't need legacy C++ footguns. It can use modern C++.

The part of Python (usually) is that you don't need to be "good at it" it you aren't trying to write super polymorphic core that runs super efficient computations like scipy. If you have a fast core engine for the innner loop, a slow Python management layer is plenty fast.

Re:We rewrote a high-performance database in Rust by dkarl on Tuesday October 18, 2022 @05:52PM (#33254535)
> Typed Python, however, scales nicely beyond this size.

Could you say more about what tools and practices make this possible, beyond simply adding type annotations in your code? Asking for a friend.

> It's tempting to split a project into a fast "core" language, and high-level "glue" language.

I did this with C++ and Boost Python back in the day and loved the experience. I wonder if Rust will someday get a high-level language for writing applications and scripts on top of a Rust codebase, like Boost Python for C++ or Tcl for C.

Re:We rewrote a high-performance database in Rust by heavyset_go on Tuesday October 18, 2022 @06:14PM (#33254761)
> Could you say more about what tools and practices make this possible, beyond simply adding type annotations in your code? Asking for a friend.

Python with type annotations works really well with type checkers like Mypy, along with LSP servers, and both of those integrate with most development environments.

Using a Python-oriented IDE like Pycharm with type annotated Python also allows for better refactoring options. It reduces the uncertainty and guesswork an IDE's static analyzer must engage in for even basic features you'd take for granted with IDE and statically typed languages.

In practice, developers don't have to keep what can be a massively complex application running in their heads to modify code accurately. A nicely typed project makes it easy to exactly what types of data are being passed around and modified. Before gradual typing, you'd have to backtrack to all of a function's call sites to understand exactly what kind of data it takes and returns. With gradual typing, you can just look at types and rely on Mypy to ensure the right data is actually being shuffled around.

> I did this with C++ and Boost Python back in the day and loved the experience. I wonder if Rust will someday get a high-level language for writing applications and scripts on top of a Rust codebase, like Boost Python for C++ or Tcl for C.

I haven't used Boost Python, but there are some options for Rust and Python that work well and seem to suit this use case like PyO3.

Re:We rewrote a high-performance database in Rust by exceptione on Tuesday October 18, 2022 @05:57PM (#33254589)
Especially business logic should be taken in the firm grip of static compile time guarantees that the hand of a strong type system delivers. Even more so if it changes constantly! Refactoring without fear.

Only software that does not have to run correctly (prototypes, personal hobby projects) can get away with a non-static type system.

When I have to pick a tool and I see it is written in Python I will have a look for an alternative if possible. Because I know it will have many bugs: some known, lots hidden.

Re:We rewrote a high-performance database in Rust by dymk on Tuesday October 18, 2022 @05:30PM (#33254279)
It takes longer to learn how to use C++ to the same level of proficiency and correctness compared to Rust, in my experience. It's harder to write an incorrect program in Rust.
Re:We rewrote a high-performance database in Rust by lupire on Tuesday October 18, 2022 @05:48PM (#33254499)
What are the main correctness risks in C++ if you just never use a raw pointer?
Re:We rewrote a high-performance database in Rust by steveklabnik on Tuesday October 18, 2022 @06:10PM (#33254723)
I don't know about "main", but like, you don't need raw pointers to have UB. uniq_ptr is nullptr after you move it.

And even then, my understanding is that raw pointers are still intended to be used in Modern C++: they're there for when you don't want to transfer ownership.

Re:We rewrote a high-performance database in Rust by bcrosby95 on Tuesday October 18, 2022 @06:04PM (#33254662)
One thing off the top of my head, from experience:

std::string s(s);

To be fair, compilers will warn you about this nowadays. But when I converted a C codebase to C++ 20 years ago they didn't.

IIRC, references can also refer to de-allocated memory. Also, if you don't pass-by-reference or pointer, you can literally "slice" the dynamic doohickies off your instance so your AlbinoCat behaves like a Cat because all that extra special stuff is gone as far as the function is concerned.

This is just off the top of my head after not working with C++ for 20 years. I'm sure with all the new features it's gained over the past 20 years theres whole new exciting ways to blow your leg off.

Re:We rewrote a high-performance database in Rust by mamcx on Tuesday October 18, 2022 @05:56PM (#33254578)
Well...

   UNSAFE {   
      // TODO: Verify all the lines, all the time, are ok
      // Just like you do testing, documentation, security and all that
      // ok?
      #include <iostream>
      using namespace std;
      int main() {

         // YOUR CODE
      }   
   }
Re:We rewrote a high-performance database in Rust by lupire on Tuesday October 18, 2022 @05:59PM (#33254617)
What are you saying?
Re:We rewrote a high-performance database in Rust by mamcx on Tuesday October 18, 2022 @06:35PM (#33255003)
Well if the questions is:

> What are the main correctness risks in C++ if you just never use a raw pointer?

All the code on C/C++ IS a correctness "risks". Only constant, manual inspection could(maybe) say otherwise.

What Rust gives is significant reduction of the risks.

Re:We rewrote a high-performance database in Rust by maccard on Tuesday October 18, 2022 @07:24PM (#33255455)
I write c++ every day. Theres no such thing as "just never use a raw pointer", as existing code will use pointers, libraries become pointers, and there's no "non-owning" equivalent of a unique pointer. Even if there was:

- lifetimes are hard. The core guidelines recommend using span and string view, but correct usage of those isnt straightforward:

    std::string_view make(const std::string& in)
    {
        return in;
    }
Is not safe, at all. Span is also super dangerous:

    std::vector<int> vec = {1, 2};
    std::span<int> sp = vec;
    vec.push-back(1); 
Lambda capture semantics are another area where you might end up surprised by the behaviour too, (and by surprised, I mean you'll have memory issues).

Then there's all the normal stuff that still exists like slicing, lack of bounds checking, resource leaks because of improper inheritance use, data races, use-after-free issues. Sure these can be caught by a sanitizer with enough effort, but they still exist.

Re:We rewrote a high-performance database in Rust by dureuill on Thursday October 20, 2022 @01:50AM (#33271272)
how do you observe data you don't own if you never use a raw pointer or reference?

if you use shared ptr for this:

1. shared ptrs aren't made for this use case, they're for shared ownership 2. why using C++ at all if it means reducing its performance to an (atomatocally) performance counted language? Rust allows for much more performance with its safe borrowed references.

That also not what the CPP core guidelines, that are supposed to define modern C++, prescribe: For general use, take T* or T& arguments rather than smart pointers[0]. This rule incurs the risk of use after free if the returned reference is kept for too long by the caller. The rules for reference validity are non trivial (eg you returned a reference to an item of a vector, if you push into your vector you might invalidate the reference), so this is a significant source of bugs even in modern C++.

if you don't use shared ptr or references/pointers for this i'm curious as to which mechanism you're using to observe non owning data

[0]: https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines...

Re:We rewrote a high-performance database in Rust by arriu on Tuesday October 18, 2022 @05:18PM (#33254127)
I agree, rust has a difficult learning curve. I’ve often heard at least a year is required to really feel confident.
Re:We rewrote a high-performance database in Rust by gotts on Tuesday October 18, 2022 @06:14PM (#33254765)
6 months is what I heard. I'm currently at ~2 and 6 sounds like a pretty good estimate.
Re:We rewrote a high-performance database in Rust by heavyset_go on Tuesday October 18, 2022 @06:19PM (#33254803)
It takes a relatively short time to be proficient enough to make useful contributions, maybe half a year to a year to be confident. You can give an experienced developer the Rust book and have them contributing to a Rust codebase quickly.
Re:We rewrote a high-performance database in Rust by nicoburns on Tuesday October 18, 2022 @07:18PM (#33255393)
For me it was ~2 weeks to get out of "learning mode" and into actually writing production code. 3 months to feel proficient. Maybe a year to feel that I had reached parity of fluency with languages I already knew.
Re:We rewrote a high-performance database in Rust by tialaramex on Tuesday October 18, 2022 @08:07PM (#33255775)
People really vary, it felt natural to me almost immediately but Rust's surveys indicate that some noticeable proportion of respondents report not being comfortable after more than a year.

This is seen as an important thing to improve because of the slogan "A language empowering everyone to build reliable and efficient software". If Rust isn't empowering everyone because say, 10% of people who try to pick it up can't get anywhere that's not good enough.

Re:We rewrote a high-performance database in Rust by heavyset_go on Tuesday October 18, 2022 @06:03PM (#33254652)
The footgun-to-appropriate-feature ratio is higher with C++ than Rust. Rust also has some excellent Python integration options that are relatively easy to use.
Re:We rewrote a high-performance database in Rust by MisterTea on Tuesday October 18, 2022 @06:51PM (#33255128)
> What stood out for me in the article is him saying that it's difficult getting developers with experience in both Python and C++.

More like they had difficulty finding cheap experienced c++/python devs.

Re:We rewrote a high-performance database in Rust by nicoburns on Tuesday October 18, 2022 @07:09PM (#33255311)
> Is Rust such a straight-forward language that people new to the language can write a very performant programme

Yes. At least that was my experience coming to Rust from a primarily JavaScript background. My first Rust program wasn't as fast as it could have been (I know a lot more about optimising programs now than I did then), but it was still ~30x faster than the JavaScript and Pythons versions my company had attempted to write first. That's without putting all that much effort into optimising it.

Re:We rewrote a high-performance database in Rust by LarsDu88 on Tuesday October 18, 2022 @08:25PM (#33255905)
I'm 3 weeks into Rust, and I've found it easy to get over an initial hump. I think where it becomes difficult is in thinking about smart pointers like Rc, RefCell, Arc, etc -- stuff I haven't even encountered yet. For things like certain popular leetcode questions, these concepts become more valuable.
Re:We rewrote a high-performance database in Rust by SleepyMyroslav on Wednesday October 19, 2022 @10:47AM (#33263016)
>a very performant programme

You might need to define what is "very performant".

I come from C++ performance work side in games. Typical C++ is ~10 times slower than optimized C++. And optimized C++ is sometimes ~2-10 times slower than what game hardware can do. What limits many projects is that cost of going to 'next level' of performance is also nonlinear. Where is on this scale Rust code written by new to the language?