News for hackers. Stuff that matters.
Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by btilly on Thursday March 15, 2018 @02:52PM (#16595529)
I have a question about this.

Articles like http://blog.llvm.org/2011/05/what-every-c-programmer-should-... have convinced me that even if C or C++ reads logically like it is safe, there is a possibility that the compiler can rewrite your code in an acceptable way according to the standards such that the checks that are clearly visible in your code disappear, opening up the very problems that you thought you were protected against.

Is there any possibility that after an aggressive compiler gets done with inlining and optimization that that could happen here in some way? Can it be proven that if the compiler works according to the standard that this won't happen..even if the programmer accidentally trips on undefined behavior?

Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by kbenson on Thursday March 15, 2018 @04:47PM (#16596409)
The best thing I've ever encountered that encapsulated this was a Cap'N'Proto vulnerability, and the discussion here about it was enlightening as well.[1]

The highly condensed version is that the compiler optimized away an if block that was responsible for throwing an error as impossible to reach in correctly functioning code, when the whole purpose of that if block was to check that condition and error so that the program did not continue in an invalid state.

Specifically:

  word* target = segmentStart + farPointer.offset;
  if (target < segmentStart || target >= segmentEnd) {
    throwBoundsError();
  }
  doSomething(*target);
(a simplified version of the actual code) was used to detect if target had overflowed (and thus target < segmentStart). The bug report goes on to explain:

However, as it turns out, pointer arithmetic that overflows is undefined behavior under the C standard. As a result, the compiler is allowed to assume that the addition on the first line never overflows. Since farPointer.offset is an unsigned number, the compiler is able to conclude that target < segmentStart always evaluates false. Thus, the compiler removes this part of the check. Unfortunately, in the case of overflow, this is exactly the part of the check that we need.

The post that's from (the sumbmitted article to the HN discussion I linked) is fairly accessible in my view. I highly recommend reading it.

Another discussion that probably has good info is this one.[2]

1: https://news.ycombinator.com/item?id=14163111

2: https://news.ycombinator.com/item?id=14785867

Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by nobugs on Friday March 16, 2018 @12:56AM (#16598727)
-fwrapv should fix it (no warranties of any kind, batteries not included).
Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by kbenson on Friday March 16, 2018 @01:24AM (#16598822)
IIRC this particular case was only seen on a specific version of one compiler for MacOS that was provided with extra patches, so it's at first glance less of a problem than it originally looks. That said, I don't think it was doing anything illegal according to the standard (no an expert on this), and as compiled software and a library, the reach of that case may be larger than we might otherwise assume.

There are undoubtedly ways to tell the compiler to be careful, but that isn't always in control of the person writing the code, and even if some level of control exists, compilers change.

Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by nobugs on Friday March 16, 2018 @04:26AM (#16599339)
To the best of my knowledge, the only compilers to exploit overflows, are GCC/Clang (and those commercial compilers I know about, explicitly said that they are NOT going to exploit signed-overflow UB, IIRC I heard it from MSVC and xlC). And for GCC/Clang, -frapw achieves the same thing. Still, I agree that things change, but this kind of behaviour won't be easy to change (at all); OTOH, I am going to campaign to remove this UB from the standard altogether (there is no real reason for this UB, at least for the platforms 99.9% of developers are working on).
Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by EpicEng on Thursday March 15, 2018 @02:59PM (#16595592)
Perhaps someone better versed in those compilers can add to/correct me here, but I'm pretty sure that can only happen if you're invoking UB somewhere along the line.
Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by saagarjha on Thursday March 15, 2018 @03:08PM (#16595663)
Once you've entered the realm of undefined behavior, the compiler can really do whatever it likes. Before then all it can do is assume you're not doing anything undefined.
Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by masklinn on Friday March 16, 2018 @08:10AM (#16600067)
> Once you've entered the realm of undefined behavior, the compiler can really do whatever it likes. Before then all it can do is assume you're not doing anything undefined.

In practice there is very little difference and you're effectively entering the realm of UB as soon as the program starts, because through inlining and propagation a possible UB may be leveraged before the UB is sequentially hit.

Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by kolpa on Thursday March 15, 2018 @03:43PM (#16595962)
That's exactly the problem. C/C++ has undefined behavior as part of the language spec, so it can never be safe unless you use a compiler that promises to reject programs that invoke undefined behavior.
Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by staticassertion on Thursday March 15, 2018 @04:09PM (#16596145)
It's more complicated than this. C++ compilers can't just "reject" UB code - they don't have enough information at all times to prove that code invokes UB, the languages is not powerful enough to represent this in all cases.

When people say "just fix this in compilers, don't compile when they run into UB" it's a fundamental misunderstanding of the problem. Yes, in some cases the compiler can prove UB, and uses it to write optimiations, but it doesn't prove that all code is not invoking UB.

Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by gpderetta on Thursday March 15, 2018 @06:03PM (#16596879)
> a fundamental misunderstanding of the problem

No kidding! There is one right here:

> Yes, in some cases the compiler can prove UB, and uses it to write optimiation

Compilers do not prove UB , on the contrary they postulate no-UB and then use this axiom to prove other properties of the code.

Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by staticassertion on Thursday March 15, 2018 @07:02PM (#16597244)
I suppose that's true, yes. My point was more that your compiler is not hitting a line of code and saying "I know this is or is not UB" because it can not do that in the general case - it is definitely incorrect to say it is 'proving' this. Would you consider that an accurate representation?
Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by masklinn on Friday March 16, 2018 @08:14AM (#16600079)
> My point was more that your compiler is not hitting a line of code and saying "I know this is or is not UB" because it can not do that in the general case

I think it's still a (cause of) misunderstanding. For the compiler, UBs are situations which axiomatically can not occur. "This is UB" is not a concept, because the compiler assumes at all point that UBs can not occur.

The compiler doesn't go "oh you're dereferencing a pointer which may be null, fuck you", it goes "you're dereferencing a pointer so it can't be null, and thus I can remove anything assuming possible nullability".

Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by staticassertion on Friday March 16, 2018 @12:25PM (#16601699)
> The compiler doesn't go "oh you're dereferencing a pointer which may be null, fuck you", it goes "you're dereferencing a pointer so it can't be null, and thus I can remove anything assuming possible nullability".

:) This is what I'm trying to say.

Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by kllrnohj on Thursday March 15, 2018 @05:21PM (#16596626)
int get_value(int* pointer) { return *pointer; }

How is a compiler supposed to reject this for undefined behavior? It'd be absurd to demand the compiler someone knows all possible usages for get_value to find if any of them pass a nullptr, so what is it supposed to do?

And this is why the spec says things like that deref'ing a null pointer is undefined behavior, so that the compiler can take that function and do the thing you'd expect it to do and transform it into a simple memory read.

Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by fjsolwmv on Thursday March 15, 2018 @06:43PM (#16597111)
The compiler shouldn't rejected UB, it should define all behavior, and reject all code constructs that can't have behavior fully defined (like accessing raw pointers as arrays, so bounds- checking is impossible). It can throw an exception on null de-reference at runtime. That's slow, but safe.
Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by adrianN on Friday March 16, 2018 @01:47AM (#16598893)
Slow but safe is not why people choose C++ for their software.
Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by wruza on Friday March 16, 2018 @05:14AM (#16599468)
There is a compiler option for that, just run javac alias instead of g++
Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by kllrnohj on Thursday March 15, 2018 @04:09PM (#16596143)
The generally recommended thing for catching & fixing this is: https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html

CppCon has also had a lot of great talks on UB and why it is what it is, such as https://www.youtube.com/watch?v=yG1OZ69H_-o

Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by jcoffland on Thursday March 15, 2018 @09:48PM (#16597998)
This is only a problem if you are using threads or shared memory and making up your own misguided locking mechanisms or in embed code on a processor with interrupts but no locks. Normally, if you use the tools correctly you will never have to worry about compiler reordering messing anything up.

The article you linked discusses what happens when you do things you should not, like fail to initialize a variable before using it. Modern compilers, when used correctly will let you know when you do this. With the right compiler options, it won't allow you to make such mistakes.

Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by btilly on Friday March 16, 2018 @05:40PM (#16604218)
The example in the case that I linked was single threaded code having a check for a null pointer unexpectedly removed.

So no threads, no shared memory, no locking mechanisms, no interrupts. Just a compiler making valid optimization and removing a necessary guard condition.

Try again.

Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by saagarjha on Thursday March 15, 2018 @03:02PM (#16595608)
As far as I'm aware, if you stay within the confines of smart pointers (and don't drop down to the raw pointer it owns) you will never encounter undefined behavior. You may have crashes if you try to double free something, but these are defined to crash rather than letting the compiler optimize out checks.
Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by masklinn on Thursday March 15, 2018 @03:12PM (#16595700)
> As far as I'm aware, if you stay within the confines of smart pointers (and don't drop down to the raw pointer it owns) you will never encounter undefined behavior.

Deref'ing an empty (e.g. moved-from) unique_ptr is still UB.

Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by saagarjha on Thursday March 15, 2018 @03:22PM (#16595782)
Yeah, there's that. It would be nice if C++ did a emptiness check for you, but I guess that this wasn't in the cards…
Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by EpicEng on Thursday March 15, 2018 @04:44PM (#16596392)
Then you have a check on every deref, which means you incur a relatively large performance penalty for using a smart pointer. If you want stuff like this there are other languages out there.
Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by mcguire on Thursday March 15, 2018 @06:10PM (#16596920)
Would it be possible to write the check so that the null test overlaps with the rest of the instructions? If the test is anyways assumed to pass, you should only get a 1-instruction overhead, right?

(That'll only work for null.)

Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by kllrnohj on Thursday March 15, 2018 @09:27PM (#16597921)
1 instruction != 1 clock cycle. In particular that would utterly kill the usage of unique_ptr on things like small embedded processors or microcontrollers that either lack speculative execution entirely or do not have the level of branch predictor & speculative execution capabilities of a high-end x86 or ARMv8 CPU.
Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by nobugs on Friday March 16, 2018 @04:42AM (#16599379)
FWIW: in general, simpler controllers (especially those in-order ones) tend to be much more friendly to branching (exactly because they're not out-of-order). NB: I am not arguing whether unique_ptr<> should check or not: if I want checked version, I will write my own wrapper, it is not a rocket science.
Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by RossBencina on Friday March 16, 2018 @12:05AM (#16598543)
While we're on "nice to haves," it would be great if there was a way to explicitly terminate the scope of objects. Then you could have an atomic move-from-and-remove-from-scope operation. Which removes any possibility of accessing the moved-from object again.
Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by staticassertion on Thursday March 15, 2018 @04:06PM (#16596125)
> As far as I'm aware, if you stay within the confines of smart pointers (and don't drop down to the raw pointer it owns) you will never encounter undefined behavior.

No. Even with smart pointers it's possible (move out of unique_ptr, deref). But even with no pointers it's possible - index into an array without checking the bound, signed int overflow, etc.

Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by yoklov on Thursday March 15, 2018 @05:07PM (#16596527)
UAF is still possible. Iterator invalidation, and such.
Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by zach43 on Thursday March 15, 2018 @03:09PM (#16595667)
Wouldn't you have a problem with reference cycles in C++ smart pointers? Not sure if they do anything special to prevent this.
Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by nobugs on Friday March 16, 2018 @01:01AM (#16598744)
std::unique_ptr<> won't allow you to have 'owning' reference cycles; neither 'owning' reference cycles are really necessary in real-world programs.
Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by saagarjha on Thursday March 15, 2018 @03:15PM (#16595724)
std::weak_ptr is there for resolving reference cycles.
Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by pdpi on Thursday March 15, 2018 @08:54PM (#16597797)
Which means you have to design around it. You can definitely run into problems with cyclical shared_ptr dependencies
Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by saagarjha on Thursday March 15, 2018 @11:27PM (#16598368)
Well, you have to do this for every language that does reference counting.
Re:A Usable C++ Dialect That Is Safe Against Memory Corruption by pjmlp on Friday March 16, 2018 @04:21AM (#16599322)
Either that, or have a cycle collector in addition to the normal reference counting optimizations.