News for hackers. Stuff that matters.
Re:Tor in a safer language: Network team update from Amsterdam by bitexploder on Saturday April 01, 2017 @02:48PM (#14013492)
I am curious why they were advised not to use Go. Probably not a safety concern.

Edit: cgo != Go. Thanks for the responses. I have done a bit of Go, but just pure Go.

Re:Tor in a safer language: Network team update from Amsterdam by masklinn on Saturday April 01, 2017 @03:00PM (#14013557)
They were not advised against Go but against cgo. Part of what they want is incremental conversion and cgo is at the same time not-go[0], costly[1] and complex[2], and then you still need to manage the Go runtime (GC & al) from within your C system. That makes integrating the two difficult, especially when you want to replace the existing system piecemeal.

A pure-Go rewrite might be an option (in fact Tor seems pretty firmly in Go's use cases), but that's not what the Tor team is trying to do.

[0] https://dave.cheney.net/2016/01/18/cgo-is-not-go

[1] a cgo->c call is ~100 times more expensive than a go->go call, and ~400 times more expensive than a c->c or rust->c call https://www.reddit.com/r/golang/comments/3oztwi/from_python_...

[2] https://www.cockroachlabs.com/blog/the-cost-and-complexity-o...

Re:Tor in a safer language: Network team update from Amsterdam by steveklabnik on Saturday April 01, 2017 @03:37PM (#14013737)
go -> C calls have gotten way way cheaper in newer versions of Go. There's still overhead but it's not as bad as it used to be.
Re:Tor in a safer language: Network team update from Amsterdam by hckr1292 on Sunday April 02, 2017 @02:55AM (#14016211)
I love that the Rust core team is defending Go in a thread ostensibly about Rust and Tor.
Re:Tor in a safer language: Network team update from Amsterdam by riking on Saturday April 01, 2017 @04:13PM (#14013917)
It's still absolutely terrible in terms of ergonomics. You're forced to perform manual memory management, etc. I've done it a few times and I absolutely don't recommend it.
Re:Tor in a safer language: Network team update from Amsterdam by steveklabnik on Saturday April 01, 2017 @04:34PM (#14014034)
I'm not making any value judgements here; just saying that the times have decreased significantly since the links that were posted.
Re:Tor in a safer language: Network team update from Amsterdam by burntsushi on Saturday April 01, 2017 @06:54PM (#14014617)
Finalizers help with memory management in the simple cases.
Re:Tor in a safer language: Network team update from Amsterdam by danieldk on Sunday April 02, 2017 @02:51AM (#14016203)
But C code may not keep a Go pointer that persists between calls (because GC). I can imagine that this is a problem for gradually converting code bases.
Re:Tor in a safer language: Network team update from Amsterdam by burntsushi on Sunday April 02, 2017 @09:37AM (#14017126)
Sure but that's not what I was replying to? I was only talking about Go->C FFI. If someone didn't know about finalizers, then I've might be trying to insert `free` calls everywhere in their Go code, which could become quite annoying.

But yes, Go pointers in C code is bad juju.

Re:Tor in a safer language: Network team update from Amsterdam by danieldk on Sunday April 02, 2017 @10:14AM (#14017233)
Sure but that's not what I was replying to? I was only talking about Go->C FFI.

Sorry, my reply was too brief. I wanted to add that the ergonomics are bad, not just because of freeing memory (for which the inconvenience can indeed be reduced with finalizers and/or Close() methods plus defer). But rules such as this one make Go->C FFI unergonomic as well. To give one example: many linear algebra libraries (e.g. Tensorflow) have their own wrappers around raw arrays to represent tensors (with their dimensionality) [1]. As a consequence of this rule, one cannot just a pointer to the first slice element to such functions (since a pointer to a Go object would be stored in a C struct), but have to malloc an array and copy over data from the slice to the C array.

[1] There are other issues, such aligning slice memory to 16-byte boundaries.

Re:Tor in a safer language: Network team update from Amsterdam by burntsushi on Sunday April 02, 2017 @11:46AM (#14017556)
Ah yes, you are absolutely right. I actually modified Rust's regex C API in part because of this problem in Go. I can't remember the details, but they were similar to your example where the only way to work around it was an unavoidable additional allocation.

(Of course, I think the change led to a better overall API. Go just helped me get there in a circuitous way.)

Re:Tor in a safer language: Network team update from Amsterdam by geofft on Saturday April 01, 2017 @03:13PM (#14013619)
Go and Rust are very different languages. Rust is, by design, well-suited to Tor's use case, where they have a large C or C++ program and they need to incrementally rewrite parts of it (and maybe never all of it!) in a better language.

It turns out (or so I hear) that Google statically links everything in production, and has been using C++ as a language to implement HTTP endpoints for a long time. So Go is a better C++ for what they want out of a better C++; for the rest of us, it looks more like a compiled language along the lines of Python/Ruby/etc. with a nice deployment story. If you want that out of your better C++, Go is great. If you want to reimplement all of Tor from scratch, Go certainly seems like a reasonable choice.

But as a result of these priorities, Go basically doesn't have interoperability with the platform ABI as a goal. (For some combination of historical reasons and the lack of complicated features in C, the platform ABI on just about every platform these days is a C ABI.) Rust does; it uses a standard compiler toolchain (LLVM) instead of what's basically a custom one (Plan 9), and the standard toolchain knows how to generate calls that follow the C ABI. Rust doesn't have a runtime of its own, and it's safe to directly call into a Rust program from some arbitrary point in a C program. Rust's allocator doesn't care if you do stupid things with pointers it allocates, as long as you give them back eventually. Rust doesn't create threads on its own unless you ask. Rust functions use the normal stack. Rust on UNIX uses the platform libc. And so forth.

It's possible to call C code from Go and vice versa, just as it's possible to call C code from Python and vice versa. But Go is not best tool for this particular job.

Re:Tor in a safer language: Network team update from Amsterdam by JepZ on Saturday April 01, 2017 @06:11PM (#14014464)
I think Go is a better C, while C++ is the opposite.

+1 for the rest of the explanation.

Re:Tor in a safer language: Network team update from Amsterdam by socmag on Saturday April 01, 2017 @05:51PM (#14014364)
Totally agree, and Rust isn't a sane choice either even though the points you make are valid.

Rust has some interesting features, but being C isn't one of them.

Re:Tor in a safer language: Network team update from Amsterdam by geofft on Saturday April 01, 2017 @06:15PM (#14014479)
Can you expand on why you think Rust isn't a sane choice here?
Re:Tor in a safer language: Network team update from Amsterdam by socmag on Saturday April 01, 2017 @08:06PM (#14014904)
Because again you are deviating from the industry standard completely portable lingua-franca language that is designed explicitly for precisely these types of problem spaces, and is perfectly in tune with the OS and existing standard library.

What would be the advantage, just improved memory safety guarantees for people working on the project?

If that's the case start again from scratch and the first thing you do is build a handle based memory management system that everything must go through Safexxxx() versions of everything and then explicitly enforce no other memory access patterns.

If you want to do concurrency, build simple threads with input queues that would just be like channels in Go.

You can use a completely functional actor model in C. Is all this object oriented dangling pointer stuff that is scary, but there really isn't any need for that.

Software pipelines with slab allocation or ring buffers virtually guarantee no leaks and are very scalable and cache friendly.

That would be my personal recommendation.

Re:Tor in a safer language: Network team update from Amsterdam by geofft on Saturday April 01, 2017 @11:13PM (#14015557)
You can do all of these things in C - but at the point that you're enforcing use of your Safexxxx() functions, you've already given up on the industry standard. You can't use the normal OS libraries; you've got to route everything through your special functions.

The advantage of Rust is, honestly, that it has a community of people who are excited to do this sort of work in a language. At least 50% of the advantage of any particular language is not the benefits of the language directly, but the community around it (that derives indirectly from the benefits of the language, but also from marketing and other things). Some time ago I needed to write bindings to SCM_RIGHTS / CMSG_*, which is probably the trickiest part of the libc API (all of cmsg(3) is macros that do stupid things with casts). It was annoying, but it led to a better interface than C itself offers, and someone else found a bug in the OS X handling. I couldn't expect that if I were working on my own custom reimplementation of libc.

Re:Tor in a safer language: Network team update from Amsterdam by socmag on Monday April 03, 2017 @05:57AM (#14021677)
So basically what it comes down to is the argument for building it in Rust is there are some excited people.

And is funny you talk about community, since as the guys on slashdot and others pointed out, the community of people who can use and work in C effectively is orders of magnitudes more than Rust. Have you any idea how many Linux Kernel developers there are alone?

http://m.slashdot.org/story/324469

Building API's specific to the use case is part of our jobs as professional developers.

Abstracting a few things things isn't "rewriting libc", that is just general practice for most decent size projects.

Anyway, the decision is made, so the whole thing is moot at this point.

Re:Tor in a safer language: Network team update from Amsterdam by geofft on Monday April 03, 2017 @12:19PM (#14024103)
> the community of people who can use and work in C effectively is orders of magnitudes more than Rust

The key word here is "effectively." For the purposes at hand, effectiveness includes memory-safety. Do you know of a single project that implements cmsg(3) in C or C++ in a memory-safe, well-typed, cross-platform way? Or a project where I can submit a pull request and expect it to be reviewed, tested across platforms, and fixed?

I do genuinely believe that the community of people who can use and work in C effectively, in the sense of effectiveness that I and the Tor Project are interested in, is orders of magnitudes smaller than Rust.

I have a very good idea of how many Linux kernel developers there are - and also how many high-severity security bugs there are. I'm a coauthor of a research paper where we wanted to talk about exploitable security bugs in Linux, so we sat down and found a local privilege escalation in hours (CVE-2009-0024).

Re:Tor in a safer language: Network team update from Amsterdam by vertex-four on Saturday April 01, 2017 @08:40PM (#14015030)
C programs don't tend to be in tune with the OS and existing standard library if they support more than one OS. You need a heavy abstraction layer by the time you're doing anything vaguely complicated - which Tor is, by simply using an event loop. It's also not super-easy to interface with crypto libraries safely without building an abstraction layer. So you're a few hops away from being "in tune" to start.

Rust provides, among other things, a decent type system that can encode some of your invariants in a manner that's a lot clearer - and every operation goes through the equivalent of your Safexxxx() functions by default, making the actual logic of what you're trying to do further clearer. A reasonable error handling mechanism is also built into the standard library and partially into the language, rather than checking return values for -1, or is it 0, and is the actual error in errno or was that function actually meant to return a value between -1 and -100 corresponding to the error, etc etc.

Re:Tor in a safer language: Network team update from Amsterdam by hannob on Saturday April 01, 2017 @02:52PM (#14013519)
Go needs a garbage collector that needs to be set up. Using Go code within C code is possible, but it creates additional hurdles.

Therefore a slow transition of rewriting parts of the code in a safer language and having the core still in C is much less feasible with Go. With Rust you can easier just compile some object files and link them into your application.

Re:Tor in a safer language: Network team update from Amsterdam by nickpsecurity on Saturday April 01, 2017 @03:16PM (#14013635)
Here's what I'd have told them:

https://news.ycombinator.com/item?id=14013617

Re:Tor in a safer language: Network team update from Amsterdam by m-j-fox on Saturday April 01, 2017 @02:49PM (#14013498)
They were advised against cgo. In my not so recent experience, it is a huge PITA.
Re:Tor in a safer language: Network team update from Amsterdam by pjmlp on Saturday April 01, 2017 @05:00PM (#14014142)
I never understood the idea behind cgo instead of having a proper FFI like Delphi, .NET, Eiffel and so many other languages.
Re:Tor in a safer language: Network team update from Amsterdam by stepik777 on Saturday April 01, 2017 @07:53PM (#14014865)
That's probably because of coroutines.
Re:Tor in a safer language: Network team update from Amsterdam by pjmlp on Sunday April 02, 2017 @01:45AM (#14015989)
The runtime could take care of that, just like it happens in another languages.