News for hackers. Stuff that matters.
Re:Clang emits memcpy for std::swap, which can introduce und... by kazinator on Sunday April 24, 2016 @05:11PM (#11561394)
The compiler is the implementation; it cannot cause "undefined behavior".

What it can be is "nonconforming".

A correct program which hits that situation continues to have well-defined behavior (which we know from the text of the program and the standard). Just the implementation isn't handling the requirements correctly; it is not conforming.

An implementation is not bound by the standard in its own use of the standard library functions. The implementation can use a function like memcpy in ways that would be formally undefined if they occured in user code; its implementation of memcpy just has to harmonize with that use, so that the intended behavior is ensured. This is because implementations can add their own requirements to areas that the standard leaves undefined. For instance, an implementation can add the requirement to its memcpy that the copy may overlap, if the destination has a lower address. Then, the implementation can generate code which uses memcpy that way, or make internal uses of memcpy from other library functions which use it that way. It's just using its own (possibly not publicly documented) extension.

Re:Clang emits memcpy for std::swap, which can introduce und... by xorblurb on Sunday April 24, 2016 @07:35PM (#11561836)
> An implementation is not bound by the standard in its own use of the standard library functions. The implementation can use a function like memcpy in ways that would be formally undefined if they occured in user code; its implementation of memcpy just has to harmonize with that use, so that the intended behavior is ensured. This is because implementations can add their own requirements to areas that the standard leaves undefined. For instance, an implementation can add the requirement to its memcpy that the copy may overlap, if the destination has a lower address. Then, the implementation can generate code which uses memcpy that way, or make internal uses of memcpy from other library functions which use it that way. It's just using its own (possibly not publicly documented) extension.

Yes. But right now they do not do that, and it's not their implementation of memcpy they rely on, and they are exactly doing the kind of things they disallow their users to do, so the whole picture is just plain insane. They just call libc memcpy in some cases, and their rational is that "it happens to work". Except nobody is actually sure about that, it might be already buggy on PPC, and the guarantee is not documented anywhere. Completely insane - especially given their very own approach in similar situations. As usual, they are prioritizing "optimizations" over correctness, and lets not even think about defense in depth. It's a brand new level of insanity, even beyond their usual "the standard allows us to do that" excuse, because here it does not even allow that, and it seems that neither do officially all the used implementations!

Software engineers must loose the habit of taking all kind of crazy risks that can only lead to marginally faster execution of newly introduced bugs. That does not serve any purpose.

Re:Clang emits memcpy for std::swap, which can introduce und... by Dylan16807 on Sunday April 24, 2016 @05:17PM (#11561410)
The compiler has different layers. Layer X is nonconforming, which causes undefined behavior in layer M.

Layer M's memcpy doesn't have to have the same semantics as the C standard's memcpy, but the bug report implies that it does.

Re:Clang emits memcpy for std::swap, which can introduce und... by kazinator on Sunday April 24, 2016 @05:19PM (#11561415)
Yes, if "undefined behavior" doesn't just mean "a situation where the ISO language standard doesn't impose a requirement", but has a broader meaning like "behavior not defined by anyone at all".
Re:Clang emits memcpy for std::swap, which can introduce und... by Dylan16807 on Sunday April 24, 2016 @05:30PM (#11561447)
The compiler is using the ISO language standard to define the behavior of an internal piece. It is then using that piece in the wrong way. It's not generic "not defined" behavior, it is specific C "undefined behavior".

Think of that part of the transformed code as still C for relevant purposes.

Re:Clang emits memcpy for std::swap, which can introduce und... by chandlerc1024 on Sunday April 24, 2016 @06:39PM (#11561698)
I want to assure you that the compiler does not exclusively use the ISO language standard to define the behavior of internal components of the implementation.
Re:Clang emits memcpy for std::swap, which can introduce und... by Dylan16807 on Sunday April 24, 2016 @07:45PM (#11561859)
I'm talking about a specific component named memcpy. Obviously this does not apply to other components.

And even that is only if the bug report is correct.

Re:Clang emits memcpy for std::swap, which can introduce und... by chandlerc1024 on Sunday April 24, 2016 @08:18PM (#11561935)
The compiler is not producing C code that calls memcpy. It is leveraging a specific implementation of a memcpy symbol in a particular system library. The contract between Clang and that system library can include guarantees not provided by any ISO standard.
Re:Clang emits memcpy for std::swap, which can introduce und... by Dylan16807 on Sunday April 24, 2016 @08:40PM (#11561979)
Then the bug report is wrong, and disregard the second and third posts I made. They depend on the bug report not being wrong.

My first post is generic and not about this particular issue; it's still fine.

Re:Clang emits memcpy for std::swap, which can introduce und... by whitequark_ on Sunday April 24, 2016 @09:06PM (#11562059)
The bug report is not wrong. Rather, whether the current behavior is correct depends on the platform.

For example, Darwin's libc offers this contract for memcpy and clang is perfectly within its rights to generate such code (note that the IR it generates is still violating LLVM's contract on the memcpy intrinsic); glibc offers no such contract for memcpy, and so clang's code is nonconformant.

Re:Clang emits memcpy for std::swap, which can introduce und... by electrograv on Sunday April 24, 2016 @05:18PM (#11561414)
Are you saying that Clang guarantees its own memcpy implementation will be used? I don't know; but it seems odd that if this is fully intended that the compiler/linker would generate warnings from its own handling of perfectly valid C code (as shown in the bug).
Re:Clang emits memcpy for std::swap, which can introduce und... by chandlerc1024 on Sunday April 24, 2016 @05:33PM (#11561465)
No, Clang cannot guarantee its own memcpy implementation will be used. But it can be aware of which implementation will be used and make specific assumptions about that implementation in some cases.

The specific cases are when it has a genuine guarantee provided by that implementation vendor that it is OK. In the case discussed on this bug, at least one implementation provides that guarantee (Darwin). The problem is that not all provide it, and further than other parts of LLVM have started assuming that the memcpy more closely matches C semantics. At least something will have to change here, but the C semantics are really relevant at all, or what is "valid C code".

Re:Clang emits memcpy for std::swap, which can introduce und... by kazinator on Sunday April 24, 2016 @05:37PM (#11561477)
Clang is not a complete implementation without a library, right? Whoever puts Clang together with a library becomes an integrator who provides a language implementation. It's up to that integrator to ensure that the result conforms.

Clang could be also used "freestanding". Then the user might be informed: please supply your own memcpy function with such and such characteristics, since generated code needs it.

If those characteristics are just those described in ISO C, but the compiler needs something more, then that's a problem between clang and the user.

Re:Clang emits memcpy for std::swap, which can introduce und... by chandlerc1024 on Sunday April 24, 2016 @06:14PM (#11561611)
Essentially.

But I think Clang can actually do better here. The open source project is essentially providing integration with glibc and other libcs and we don't do a reasonable job of being conservative there where we don't know what requirements that layer places upon things like memcpy.

Re:Clang emits memcpy for std::swap, which can introduce und... by MaulingMonkey on Sunday April 24, 2016 @06:47PM (#11561718)
> The compiler is the implementation; it cannot cause "undefined behavior".

The compiler is only part of the implementation. So is the standard library. So would be a processor. An implementation as a whole, perhaps, cannot be said to cause "undefined behavior", but I think it's correct to say it's parts certainly can.

If a preprocessor took valid C, and emitted valid C that could be fed into any other complete freestanding C implementation, would it not be accurate to say that this preprocessor introduced undefined behavior if it replaced, say, memmove(a, a, s) with memcpy(a, a, s) - sans any guarantees about which compiler it would have the emitted C fed into?

When it's LLVM IR invoking glibc's memcpy instead of a memmove, why is that not "causing undefined behavior"? Because it's not a C program? And if I misinvoke it via C++, is it only undefined behavior because the C++ standard references the C one? What if I use C# to invoke memcpy, which doesn't?

> An implementation is not bound by the standard in its own use of the standard library functions.

This is all well and true from a theoretical sense, but an implementation is bound to it's own standards as a means of achieving a conforming compiler, and in this case those standards are clearly documented to mirror the standard's standards. These standards are being violated, leading to possible future misbehavior when combined with other optimizations which may leave the code behaving in indeterminate ways - and, for that matter, current misbehavior in the form of triggered "false" positive valgrind checks.

> The implementation can use a function like memcpy in ways that would be formally undefined if they occured in user code; its implementation of memcpy just has to harmonize with that use, so that the intended behavior is ensured. This is because implementations can add their own requirements to areas that the standard leaves undefined. For instance, an implementation can add the requirement to its memcpy that the copy may overlap, if the destination has a lower address. Then, the implementation can generate code which uses memcpy that way, or make internal uses of memcpy from other library functions which use it that way. It's just using its own (possibly not publicly documented) extension.

It can do these things, if the implementation as a whole is coordinated in this way, but I would hope for sanity's sake it would not. And in this case the implementation as a whole has not been coordinated in this way.

Invoke copy_forward from memcpy if you like, where the former has fewer preconditions - but to write code that relies on an implementation's memcpy having fewer preconditions than the standard, even if you control the entire implementation, sounds untenable. Someone will forget that your memcpy has fewer preconditions, and write optimizations, debug checks, etc. which misassume additional preconditions that are outlined by the C++ standard. They will also also accidentally write code assuming fewer preconditions in another context where they're not guaranteed some specific implementation's relaxed preconditions.