> Rc is common in the Rust ecosystem

I'm really surprised you think that's the case. Most rust codebases I've worked with use Rc very sparingly, if at all. If a codebase does use Rc there's usually one central thing that is Rc'd, with everything else using regular memory management.

I have noticed that beginners coming from GCd languages often tend to structure their code in such a way that paints them into a corner where they must use Rc. This might be what hit you. I'm not really sure how to teach idiomatic Rust though.

> there's usually one central thing that is Rc'd,

Right.

In my case, I was using the nphysics library, which for unsurprising reasons uses Rc for entity handles.

Similarly, in my app's code, I used nphysics' proximity and contact handlers to watch for events that indicated a change in status. Those handlers needed to initialize/set values in a HashMap accessible (and periodically cleared by) the main sim loop -- 'one central thing' that is Rc'd.

It's not sprinkled everywhere in your codebase, and for obvious reasons most libraries don't need to use it. But in applications, a central loop + library-provided callbacks + some Rc'd state of doesn't seem too uncommon.

> I'm not really sure how to teach idiomatic Rust though.

Learners should read and write lots of Rust. That's a great way to learn what is idiomatic. There will be early missteps, but thankfully the language makes it more comfortable when you do things the right way.

Yeah, my advice in general is to pick a few rust codebases and start hacking on them.

But I would love to figure out how to teach this in a way that doesn't involve random codebases. Some folks learn by doing (I do!), but others fare better with tutorials.

> Just my 2p for others learning: for me, Rc::RefCell was what I was missing, even after I thought I was up to speed.

I started (and then stopped) learning Rust a few months ago (before their docs rewrite) and the borrow checker and concepts weren't explain in a way that I could understand. Programs wouldn't work at all, and when I read about Rc::RefCell I was scared because I wasn't sure how much garbage collection rust would do and whether or not it would actually be safe.

So yeah. Rust definitely has problems for beginners.

To be clear, Rust has no garbage collector, and unless you type "unsafe", it's safe. RefCell will panic when something goes wrong.

I should take this opportunity to mention that the book has been stagnating because I (along with Carol Nichols || Goulding) have been re-writing it out of tree: https://doc.rust-lang.org/stable/book/ The ownership/borrow checker stuff has been completely re-done. The existing book chapters explain it in the abstract; in the new book, we use String/&str as a motivating example, since that's an area that often trips up new Rust programmers.

That's great to hear. That's definitely an area that has made me slow down on learning Rust. Will definitely be checking out the new book once it's ready.

> I wasn't sure how much garbage collection rust would do

Rust doesn't do magical garbage collection.

Rc<T> does reference counting, which is a form of garbage collection, but you get to choose where it gets applied, so it's a linear cost with no magical GC pauses or whatever. Rc isn't unique to Rust, it exists in C++ too.

RefCell makes mutation within an Rc safe. It panics if you misuse it.

http://manishearth.github.io/blog/2015/05/27/wrapper-types-i... has more on the Rc<RefCell<T>> pattern

Reference counting is not a form of garbage collection. Both reference counting and garbage collection are types of automatic memory management.

This discussion crops up each time reference counting is brought up.

In academia reference counting generally falls under the umbrella of GC. In the industry "GC" usually means "tracing GC".

The terminology is irrelevant to the point I'm making. I did make a distinction between RC and the "regular magical kind" of GC.

Garbage collection requires a garbage collector, which reference counting doesn't have or need. This is true for both academia and industry.

It's still runtime memory management. That makes it garbage collection in my book. Sure it's not as involved as tracing, the prototypical form of GC, but it's still something that requires a runtime check each time that resource is used or a binding to it goes out of scope, similar to aspects in generational GC.

The point is that while both reference counting and garbage collection are types of automatic memory management, the reverse isn't true.

From a 1976 paper on automatic memory management https://www.cs.purdue.edu/homes/hosking/690M/deutsch.pdf

"Automatic reclamation of storage no longer in use is done by the following two techniques:

* Garbage collection * Reference counting"

Note that these are two separate items and that one is not a subset of the other (and vice versa). It is simply incorrect to call reference counting "garbage collection" when the academic and industrial practices already have explicit meanings to these two terms.

And http://dl.acm.org/citation.cfm?id=356854 (pdf: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.468...) treats reference counting as a "way to distribute GC overhead time".

I'm not saying the term always is used that way in academia. I'm saying that it's sometimes used that way, and that it's valid to call RC a form of GC.

Ironically I said that RC was a form of GC to avoid precisely this argument, because if I say "Rust doesn't have GC" I'll have a bunch of folks telling me that RC is a form of GC. This is largely irrelevant to the point I was making, which explicitly distinguished between RC and tracing ("magic") GC -- which I suspected (but wasn't sure) was what was being asked about.

> So yeah. Rust definitely has problems for beginners.

IMO Rust has the challenges that are very similar to all other languages have. But if you are trying to make the leap from a GC'd language like Java or Python to Rust without ever having written C/C++, you should expect to have to learn not only new language concepts but new programming concepts.

I'm a programmer that has been using C for at least 8 years, and I've been developing in Go for the past 4 (as well as a couple of years of Python here and there). The point is that I have experience in quite a few programming languages, and I am familiar with the programming concepts you're referring to.

I do not agree that Rust's challenges are the same challenges with other programming languages. Because Rust will not let you even play with the langauge unless you understand how to write safe programs (which is a concept that is defined in Rust). So there's a whole bootstrapping problem of "how the hell do I play with this thing to understand it if I can't play with it until I understand it fully". C, C++, Python, Go -- none of them have this issue. They will let you write bad code and won't stop you from running it (which I admit is not a good thing, I'm just saying that Rust's strengths are not without their downfalls).

I had a similar problem with the type system in Haskell. I'd look at Haskell code and it was obtuse to me. Even reading books, it was hard for me to understand some stuff until I started writing code... but writing code was difficult because I didn't understand it.

For me the key was just doing toy problems over and over again. Without understanding the idioms, it's hard to dive in and write something real. I think Rust is approachable if you take it like that (or at least it was when I looked at it about 2 years ago).

Reference counting can be a useful memory management technique. But reference counting may cause unpredictable pauses when large amounts of objects suddenly need to be freed (e.g. when dropping the last reference to a large array holding many references).

At least a carefully written garbage collector can free objects incrementally, and concurrently.

So memory management in Rust is certainly not a solved problem.

EDIT: Removed mention of RefCell.

> So memory management in Rust is certainly not a solved problem.

This looks like nay-saying just for the sake of it, but in case you're actually just confused:

Rust isn't reference counted unless you yourself add reference counting. Rust tracks ownership in the compiler so it knows statically, at compile time, when an object needs to be freed, and the compiler emits the code to do the freeing in that spot. Rc is a utility function: you can choose to opt into reference counting on a per object basis. But if you're not actually typing Rc yourself then it's not happening. Rust is no more reference counted than the ability to build the same thing in C makes C a reference counted language.

> But reference counting may cause unpredictable pauses when large amounts of objects suddenly need to be freed (e.g. when dropping the last reference to a large array holding many references).

Then again so does manual memory management if you're freeing a large object graph at once.

> At least a carefully written garbage collector can free objects incrementally, and concurrently.

You could get that with refcounting, stashing Rc0 objects in a list of items to free incrementally rather than freeing it all at once and synchronously.

Since refcounting only happens expicitly it should be pretty clear where possible drops happen that lead to large pauses.

It wouldn't be too hard to replace that Rc with an Rc that performs deferred drops by putting them in a queue and performing a number of drops at an oppurtune time (eg. using an idle-hook in a event loop).

It might even be possible to send all drops to a thread dedicated to dropping, but that would probably only work if the contents are Sync or Send. This form of async drops should always be possible for Arc.

But big pauses should rarely occur since refcounting is something that has to be added explicitly, Rust is a language where compound types are used extensively instead of building dynamic structures using collections for trivial things and the use of Drop trait is not something that can be fully relief upon (making that most solutions try to go without using it). This should make most drops pretty lightweight, even when dropping large numbers of items.

> But reference counting may cause unpredictable pauses when large amounts of objects suddenly need to be freed (e.g. when dropping the last reference to a large array holding many references).

That's still a predictable pause: it's the number of references contained within the array. If you know your array will hold at most ten references, it will take at most the time to free the ten references. Also, you know this will happen when the last reference is dropped - no earlier, no later. If your program can't handle the pause at that moment, you hold the reference until it can. If you know another component (perhaps in your own call stack) holds a reference, you know it won't be freed. And so on.

> If you know your array will hold at most ten references, it will take at most the time to free the ten references.

True. But if those references themselves contain references, it may become tricky to manage all of this. You basically don't want to think about it.

But, like others mentioned, you can put the objects in a queue, and free them in the background. It would be interesting to know how such a solution stacks up, performancewise, against an incremental, concurrent garbage collector.

Note that you don't need a GC to do that; the allocator can do it for you.

A lot of folks think that optimizations like having allocation arenas and spreading out deallocation pauses are unique to garbage collectors, but they're completely orthogonal to GC. You can have GCs with these optimizations, and GCs without. You can have regular allocators with these optimizations, and regular allocators without. Jemalloc does have arenas and stuff for allocation (I'm unsure if it spreads out deallocation loads). Of course, with a GC you can also defer the cost of iterating through large vectors and calling destructors.

But anyway, this only becomes a problem when you have large complicated Rc-trees in your application, which tends to not be the case in Rust.

An exception (large complicated Rc-trees) is the rope in xi-editor. If you load a very large file, the operation of letting go of the last reference is potentially a large enough pause to have an effect on UI responsiveness. I've considered adding a mechanism that moves the object into a deallocation thread (or a deallocation task running in a thread pool) for this reason, although I'm not sure how important it is in the grand scheme of things.

In a GC'ed language such as Go, this would not be a problem. The flip-side is that I can safely and efficiently do updates in place, because Rust's reference counted type has a way to determine when you're holding the only reference.

> I've considered adding a mechanism that moves the object into a deallocation thread (or a deallocation task running in a thread pool) for this reason, although I'm not sure how important it is in the grand scheme of things.

The one time I've done this in C++, delayed deallocation actually made major loads/unloads worse for us on mass batch operations - more cache thrashing perhaps? Since this involved GPU resources, perhaps some bad interaction with the driver? I did keep around the "optimization" conditionally for smaller operations as this let us get rid of some hitching.

> In a GC'ed language such as Go, this would not be a problem.

In C# this traditionally manifested as lengthy GC pauses you had to jump through hoops to workaround. Modern GCs are much better these days, but it's not 100% solved.

Hmm, yeah cache thrashing sounds a likely culprit - agreed. Perhaps before dropping all the references they should be sorted by memory address, so that page faults are minimised.

I'm hoping for something like this a few years down the line:

http://manishearth.github.io/blog/2016/08/18/gc-support-in-r...

Allowing Rust to generically adapt to an arbitrary Gc that it is nested inside of would be awesome.

Yes, that would be amazing.

To be clear, RefCell does not do reference counting, Rc does. RefCell does runtime borrow checking.

>Reference counting can be a useful memory management technique. But reference counting may cause unpredictable pauses when large amounts of objects suddenly need to be freed (e.g. when dropping the last reference to a large array holding many references).

How is it compared to other forms of GC?

More performant, generally, particularly considering the fine-grained control it offers. However, cyclic datastructures may be problematic (similar to how they are in typical GC, but often worse). Of course, exceptions abound; the "better" choice is extremely use-case dependent.

It's generally predictable pauses, fwiw.

Well, a different kind of unpredictability from other GCs.

And Rc is rare enough (IME) that this doesn't usually matter.

What kind of application s would Rust be an appropriate for? What languages is it largely meant to replace/improve on.

I've been using Rust to process fairly large amounts of streaming financial data. I am using Protocol Buffers (and evaluating Cap'n Proto) for network serialization and Kafka for buffering/queuing, and overall the library support is quite good.

Previously I was using JVM languages for this purpose, but grew weary of the resource footprint, and especially the unpredictable GC pauses. I am aware of the Azul JVM which removes GC pauses and of various Java techniques to avoid GC altogether, but switching to Rust provided a GC-less model from the ground-up, a powerful type system, and familiar functional programming facilities at no cost.

What Rust libraries are you using for protobufs?

I have been using https://crates.io/crates/protobuf. It generates pure Rust protobuf implementation files.

How easy is it to get Kafka and Rust working together?

Fairly straightforward overall using https://crates.io/crates/kafka. The client doesn't automatically handle Kafka node failures, so that's the responsibility of your application code.

I haven't had the chance to try https://github.com/fede1024/rust-rdkafka yet, but it looks promising and partially wraps the C/C++ library https://github.com/edenhill/librdkafka.

> What languages is it largely meant to replace/improve on.

I think it's great for any problem area where you'd instinctively reach for C. System utilities, bare metal development, etc. Stuff where you care about the precise layout of memory but would prefer that a simple but non-obvious mistake didn't end up as a high-profile CVE.

> What languages is it largely meant to replace/improve on.

C. It has C's straightforward machine model in mind, like C its memory behaviours are entirely predictable, it's entirely explicit about error handling (no hidden paths of errors exiting functions as in C++).

It improves on C by adding strict checking to make managing memory safely and avoiding race conditions tractable problems.

Here at ThreatX[1] we use it as a replacement for C. We've used it to write our web application firewall sensor and a real time threat analytics engine. Compared to C it has enabled us develop features rapidly and safely without sacrificing any performance.

1. https://threat-x.com/

It's meant as a c/c++ replacement.

Rust is intended for applications where you need speed, concurrency, and safety/correctness. In some senses, "use Rust where you would use C or C++" makes sense, but we're also seeing a lot of usage from programmers who have rejected C or C++ for various reasons.

One angle that we've been focusing a lot on lately is productivity. Think about some feature of Rust that provides safety, like out borrow checker, which ensures that pointers don't do bad things. One way to think about this is safety, but another way is productivity: if your application segfaults, you have to track down what actually caused it, and then what caused the cause: "oh this pointer was null because I did something incorrect in this other part of my code." While having the compiler do these checks can sometimes feel like a productivity _loss_ at the beginning, you do a lot less debugging later, so it's a productivity _gain_ overall. (Or at least, we feel that way.)

Other features of Rust make it feel productive as well: a focus on iterators and composable iterator adapters is often more productive than manual for loops, Cargo and crates.io enable wide-spread code re-use[1], we've been working on good tooling for IDE integration for those that use IDEs, and "zero-cost abstractions" help you have nicer interfaces while not having to pay the cost for them. Like this: https://news.ycombinator.com/item?id=13117608

One interesting area where we've seen lots of production Rust usage is embedding Rust in other languages. Since Rust can expose functions that look like C to the outside world, you could write a Ruby, Python, Javascript, or whatever extension in Rust instead of C. And Rust's safety features are appealing here, since you may not be the kind of person who writes C all the time: that's why you write Ruby/etc in the first place.

Soon, we expect to see more usage of Rust on the server: the "tokio" project is gearing up for an initial release, which provides a foundation for asynchronous IO. Even before tokio is released, people are playing with this: crates.io is Rust on the backend, and "npm recently began replacing C and rewriting performance-critical bottlenecks in our registry service architecture with Rust": https://medium.com/npm-inc/npm-weekly-73-no-love-for-http-ur...

Basically, lots of places. We'll see how things go into the future!

1: Anecdote time: my favorite package is https://crates.io/crates/x86, which provides low-level bindings to various x86 platform details. Writing an operating system? No need to define your own IDT entries, just grab the library and http://gz.github.io/rust-x86/x86/irq/struct.IdtEntry.html has you covered. Re-usable packages in operating systems is super cool. On a higher level, this kind of thing is enabling Firefox to re-use Servo components more easily; Firefox can just pull chunks of servo with (relative) ease.

Dream time, Microsoft gets to sponsor Rust on their stack and VS integration, and we can move on from C# + C++/CX to C# + Rust. :)

I did see this go by a while back: https://github.com/Microsoft/BashOnWindows/issues/258#issuec...

So someone over there is paying some degree of attention, at least :)

Have you tried VSCode with the Rust(racer,rustfmt) + lldb integration? I've been pretty impressed with it so far.

FWIW C# <-> Rust integration is really straightforward. You can actually pass delegates as C fn pointers and then treat them as a closures in Rust. Much less painful that I initially thought.

It is not yet the same as Blend + Visual Studio (C#, F#, C++/CX, C++/CLI).

Yes, I do use VSCode, but only for dabbling on Rust during plane/train travels. The language is not yet at a level it just fits on MS stack and is requested by our customers on their Requests For Proposals.

Regarding C# <-> Rust interoperability, it is very badly documented. I gave up on searching for it, and just used C# <-> C++/CX <-> Rust instead.

Or I am very bad searching for it.

Have you seen the FFI omnibus http://jakegoulding.com/rust-ffi-omnibus/ which includes C# examples?

No, thanks for pointing it out.

C# <-> Rust is pretty straightforward. Just follow the C FFI side on Rust and C# has standard marshal mechanisms for calling C code.

extern/dllimport[1] Covers most of it. There's automatic conversion for CString/string and delegates as function pointers. If you need to go deeper than that there's the marshal namespace[2]. Going through C++/CX sounds really painful.

[1] - https://msdn.microsoft.com/en-us/library/e59b22c5.aspx

[2] - https://msdn.microsoft.com/en-us/library/system.runtime.inte...

> C++/CX sounds really painful.

Not for someone that knows C++ since C++ARM. :)

I am pretty comfortable with C# and native interop, my issue was trying to map Rust strings with .NET UTF-16 ones, including passing ownership from Rust to .NET side.

If I remember right you can specify encoding as an attribute on the extern decl.