News for hackers. Stuff that matters.
Re:Rust, Lifetimes, and Collections by Animats on Wednesday November 19, 2014 @03:06PM (#8632290)
Anyone calling foo3 will try to index beyond the end of the array, and all hell will break loose. What do you do then? Well, you grep for the word "unsafe" and you start auditing unsafe blocks, looking at all the inputs that you send into them.

That's not a very good solution. "Kill problems, don't just wound then" - Kelly Johnson, Lockheed Skunk Works, designer of the U-2 and SR-71.

I used to work on this problem, about a decade ago. I was trying to figure out how to make C++ safe. I finally gave up, since memory safety wasn't a priority with the C++ committee. I'd proposed scope-based "borrowing" (I used the term "keeping", as the inverse of borrowing) and reference counts.

Complex data structures remained a problem. The best idea I could come up with was to use weak pointers heavily. When you have something like a GUI widget library, with pointers going upwards, downwards, and sideways, who owns what is a big issue. One solution is that all the graphics widgets are owned by some top level object, like the "window" object, all the inter-widget links are weak, and all the weak pointers are in objects that can't outlive the top level object. This creates a sort of allocation pool. Objects pointed to by weak pointers can't fully go away until the parent object of all the weak pointers goes away. They only get marked as invalid for weak pointer referencing.

This is memory-safe, but you don't get the space used by widgets back until the top-level page object is deleted. That can be a problem for long-lived graph structures with a lot of churn, like the scene graph in a game or the active collision structures of a collision detection system. For most applications, though, it's not a big deal; browsers build a DOM, use a DOM, and discard a DOM when the page closes. Might be a problem if you left Facebook open for a month.

Go for safety. We do not need yet another unsafe language. We have plenty of those. The day there's a CERT advisory for a buffer overflow in a Rust program, Rust has failed.

Re:Rust, Lifetimes, and Collections by kibwen on Wednesday November 19, 2014 @03:23PM (#8632383)
Fortunately for your use case, Rust provides a reference-counted type in the standard library that does indeed feature weak pointers to break cycles. It's like C++'s `shared_ptr`, except safer and faster (since Rust's safety rules mean that we can prove that the refcounted data isn't shared between threads, and hence we don't need to resort to atomic operations). All of these are possible because of a thin layer of `unsafe` code on the ground, which we then wrap in a safe interface.

  > We do not need yet another unsafe language. We have plenty of those.
I sadly must disagree. If safety were truly paramount, people would have dropped C++ entirely in favor of Java in the 90s. The people still using C++ aren't idiots, they just have requirements with respect to efficiency, control, determinism, and expressiveness that other languages don't match. Rust exists to provide an alternative to these people, so that we can at last bring some semblance of safety to domains with the requirements intimated above.

  > The day there's a CERT advisory for a buffer overflow in a Rust program,
  > Rust has failed.
I dread that day, and yet I know that it's inevitable. In the meantime, that same buffer-overflowing Rust program may well have produced ten times the CERT advisories had it been written in C. I'm not an absolutist; I'll take that improvement as a win.

  > I used to work on this problem, about a decade ago.
  > I was trying to figure out how to make C++ safe.
Honestly, the Rust developers would love your perspective if you'd like to take a harder look at Rust.
Re:Rust, Lifetimes, and Collections by Animats on Wednesday November 19, 2014 @04:20PM (#8632686)
Well, one key issue is that the compiler needs to know that program-terminating fault checks are special for optimization purposes. It's OK to terminate a program early. If a loop, when entered, will inevitably subscript out of range at iteration 101, then it's OK to detect that at loop entry, and never execute iterations 0..100. Even if there's a print statement in the loop.

Normal compiler hoisting operations aren't allowed to do that. It violates normal execution order semantics. If you don't do that, you can't optimize checks fully. This is why generating assert statements in macros or templates and then trying to optimize them doesn't work. Asserts may need to be hoisted through statements with side effects. Compilers have to handle that at the flow graph level.

This works best if you're not allowed to capture such errors. If you're headed for an unavoidable crash, it's best to have it sooner, rather than later. Go takes a hard line on this - no exception mechanism. (Admittedly some people are abusing Go's "panic/recover".) Google's position is that in their kind of operation, it's better to fail the whole program. They can recover from a failed server, after all. Erlang also take a "fail early" position.

As for the need to eliminate errors, the recent history of computer language development has been in the direction of increased safety. All the major scripting languages are memory-safe. Newer hard-compiled languages such as D and Go are memory-safe, although they have to use GC to do it. A whole generation of programmers have never had to look for a buffer overflow or a dangling pointer.

Meanwhile, in the security world, the buffer overflow threat has become worse. It's not script kiddies any more. It's national intelligence agencies (ours, theirs, who knows who else?) with big budgets and competent people looking for exploits. There's a black market in zero-day exploits now. Rust has a fighting chance to fix that. Not just trim it back a little. Fix it. Please don't blow this chance.

Latest buffer overflow reported by CERT: November 14, 2014:

https://www.us-cert.gov/ncas/alerts/TA14-318B

The Microsoft Windows OLE OleAut32.dll library provides the SafeArrayRedim function that allows resizing of SAFEARRAY objects in memory.... In certain circumstances, this library does not properly check sizes of arrays when an error occurs. The improper size allows an attacker to manipulate memory in a way that can bypass the Internet Explorer Enhanced Protected Mode (EPM) sandbox as well as the Enhanced Mitigation Experience Toolkit (EMET).

This vulnerability can be exploited using a specially-crafted web page utilizing VBscript in Internet Explorer. However, it may impact other software that makes use of OleAut32.dll and VBscript. Exploit code is publicly available for this vulnerability. Arbitrary code can be run on the computer with user privileges.

That's why "carefully checked" "unsafe" code doesn't work. "SafeArrayRedim" - wasn't.

Re:Rust, Lifetimes, and Collections by dbaupp on Wednesday November 19, 2014 @04:33PM (#8632741)
> All the major scripting languages are memory-safe. Newer hard-compiled languages such as D and Go are memory-safe, although they have to use GC to do it. A whole generation of programmers have never had to look for a buffer overflow or a dangling pointer.

They all have the ability to write unsafe code in the same manner as Rust; but only Rust has it really built-in to the language so the compiler can reason about it. E.g.

- http://golang.org/pkg/unsafe/

- https://docs.python.org/3.4/library/ctypes.html

It just doesn't make sense to presuppose that it is any worse in Rust. In fact, quite the opposite, the type system is designed for ensuring memory safety without a GC, and in multithreaded environments. I believe this is not true for Go (it is vulnerable to data races, and requires a runtime tool to detect some subset of the ones can occur in any given program), but I don't know about D.

Rust cannot remove `unsafe` without being useless for its target, and useless as a language (it wouldn't even have FFI).

Re:Rust, Lifetimes, and Collections by Animats on Wednesday November 19, 2014 @06:56PM (#8633559)
Calling external code is a separate issue. This is about unsafe code within the language.
Re:Rust, Lifetimes, and Collections by dbaupp on Wednesday November 19, 2014 @07:16PM (#8633638)
I thought it was about buffer overflows and security vulnerabilities within Rust programs? i.e. not just the parts written directly in Rust. Looking at anything else seems a bit silly, since the thing that runs in the real world is the whole program.

The low-level code has to exist somewhere, vectors need to be allocated, strings need to be printed, and people will want the performance. This code is either going to be in external libs called via FFI, as raw codegen in the compiler or in Rust. The last sounds like the one that is safest since the FFI code would have to be written in one of the unsafe systems languages, while writing in Rust gives all the type system advantages and high-level features (references, pattern matching), not to mention avoiding the (mental) overhead of correctly interacting between languages.

It seems to me that taking away the low-level power of opt-in `unsafe` is just myopically punting the problem along, rather than seeing it as giving Rust the ability to avoid having to write really unsafe C plugins (like Python etc.).

Re:Rust, Lifetimes, and Collections by kibwen on Wednesday November 19, 2014 @07:19PM (#8633645)
But they aren't separate issues. If you couldn't do these things in Rust, people would just drop into C to do them. Given that that would introduce much more unsafety into Rust programs, Rust does the right thing by reifying unsafe memory operations and sequestering them away.
Re:Rust, Lifetimes, and Collections by Dewie on Wednesday November 19, 2014 @06:16PM (#8633331)
> Rust cannot remove `unsafe` without being useless for its target, and useless as a language (it wouldn't even have FFI).

I like to think of Haskell as a parallel: It guarantees side effect-free code. But you still need an escape hatch in the form of unsafePerformIO in order to use the FFI. The language's semantics can easily be broken with that functionality, but it seems to be a necessary price to pay if you want to have any kind of FFI. (<- Well, I guess something like Proof Carrying Code could be an alternative..)

Re:Rust, Lifetimes, and Collections by kibwen on Wednesday November 19, 2014 @04:58PM (#8632898)
I'd like to reassure you that the Rust community absolutely does not condone wanton usage of `unsafe` unless profiling indicates an area that really requires low-level trickery. The discussion in the article linked here concerns the standard library, which needs to be fast because otherwise people will reimplement it themselves, likely poorly, over and over again. By accepting potential unsafety here we are saving others the effort of a half-baked, insecure collections library while also concentrating our audits on well-known and oft-used code.

  > All the major scripting languages are memory-safe.
  > Newer hard-compiled languages such as D and Go are memory-safe
Rust's definition of safety actually surpasses most of these languages, as it extends to safety in the face of concurrent operations. Rust statically prevents data races, which AFAIK is a property that until now has only been exhibited by thoroughly immutable languages such as Erlang. That it manages all this without a GC is very exciting.

  > Meanwhile, in the security world,
  > the buffer overflow threat has become worse.
The Rust developers agree with you. Rust is motivated by the fact that Mozilla is tired of Firefox getting pwned by dint of C++'s looseness, and despairing at the fact that further improvements to Firefox will entail tacking concurrency onto a multi-million-line C++ codebase dating back to 1997, which would be notoriously difficult even if security weren't a concern. They're writing a brand-new browser engine in Servo with a focus on massive concurrency and user safety.

I guarantee you that Rust cares a great deal about safety. It goes to lengths that almost no other language is willing to go in pursuit of this goal. However, it is also a thoroughly pragmatic language. We won't make the world a better place by blindly preaching dogma. Rust also won't be the last language ever made. If nothing else, even baby steps toward a safer world are a measure of progress. :)

Re:Rust, Lifetimes, and Collections by dbaupp on Wednesday November 19, 2014 @04:17PM (#8632677)
Rust has to have some equivalent to `unsafe`; it is designed to be used for low-level code, and opt-in unsafety is far better than opt-out. Anything like FFI is outside the realms of what a compiler can possibly verify, since it knows nothing about what the arbitrary C functions do, or what pre-/post-conditions hold. Furthermore, all the normal tools provided by Rust are available in unsafe blocks.

The low-level code has to be written somewhere (Rust needs to have a vector type, needs to have an efficient hashmap), and frankly, writing it in Rust is better than writing it in C and using FFI to import it.

Rust is going for safety.

Re:Rust, Lifetimes, and Collections by Dewie on Wednesday November 19, 2014 @06:11PM (#8633311)
> Go for safety. We do not need yet another unsafe language. We have plenty of those.

Do we? The only ones I hear about people using are C/++. On the other hand, safe systems languages seem to end up being forgotten, as people opt for C/++ instead for systems/performance domains. Presumably in part because they use things like garbage collection, giving up some of the more raw control and probably also speed that C/++ provides.

It really seems that any language that wishes to challenge C/++ on their own turf needs to offer the power of C/++ as far as it is possible. In Rust's case, that is possible as long as you're being explicit about where the safe interfaces end and the unsafe ones begin.