No, what you described is implementation-defined behavior.
It may be confusing, but here's the breakdown of different kinds of behavior in the C standard:
* Well-defined: there is a set of semantics that is defined by the C abstract machine that every implementation must (appear to) execute exactly. Example: the result of a[b].
* Implementation-defined: the compiler has a choice of what it may implement for semantics, and it must document the choice it makes. Example: the size (in bits and chars) of 'int', the signedness of 'char'.
* Unspecified: the compiler has a choice of what it may implement for semantics, but the compiler is not required to document the choice, nor is it required to make the same choice in all circumstances. Example: the order of evaluation of a + b.
* Undefined: the compiler is not required to maintain any observable semantics of a program that executes undefined behavior (key point: undefined behavior is a dynamic property related to an execution trace, not a static property of the source code). Example: dereferencing a null pointer.
§3.4.0: behavior: external appearance or action
§3.4.1: implementation-defined behavior: unspecified behavior where each implementation documents how the choice is made. EXAMPLE: An example of implementation-defined behavior is the propagation of the high-order bit when a signed integer is shifted right.
§3.4.2: locale-specific behavior: behavior that depends on local conventions of nationality, culture, and language that each implementation documents. EXAMPLE: An example of locale-specific behavior is whether the islower function returns true for characters other than the 26 lowercase Latin letters.
§3.4.3: undefined behavior: behavior, upon use of a nonportable or erroneous program construct or of erroneous data, for which this International Standard imposes no requirements. NOTE: Possible undefined behavior ranges from ignoring the situation completely with unpredictable results, to behaving during translation or program execution in a documented manner characteristic of the environment (with or without the issuance of a diagnostic message), to terminating a translation or execution (with the issuance of a diagnostic message). EXAMPLE: An example of undefined behavior is the behavior on integer overflow.
§3.4.4: unspecified behavior: use of an unspecified value, or other behavior where this International Standard provides two or more possibilities and imposes no further requirements on which is chosen in any instance. EXAMPLE: An example of unspecified behavior is the order in which the arguments to a function are evaluated.
[1]: WG14 working paper for the C11 standard: http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf
If it were otherwise, you couldn't `memcpy()` structures around.
I'll check the unbelievably thick book that tries to specify C11 (I've printed it, it's over 2 pounds).
Why not just have the compiler zero the memory, and thereby remove the trap? Seems very sensible to me. Do you think it’s a bad idea, and if so, why?
As a type qualifier keyword:
{ int x, y; /* x and y are zero */ }
{ int noinit x, y; /* x is indeterminate, y is zero */ }
{ noinit x, y; /* both x, y indeterminately-valued */ }
{ int x, y = noinit; /* x zero, y indeterminate */ }
decl (unspec_order) { /* comma-separated list of decl items */
a[i] = i++; /* UB */
}
a[i] = i++; /* well-defined */ static int x, y; /* x and y are zero */Oh no no no, I was picking on hermitdev's characterisation of the behaviour. Sure, what the compiler does with the padding bytes is unspecified. But it can still lead to undefined behaviour, if some unwary user ever reads them. A misinterpretation like that, and poof you get a security vulnerability. The C standard is insane.
Modern processors, with their out-of-order execution, complex caching algorithms, deep pipeliens and multi-threaded hardware often render this sense of control more illusory than factual, but the C culture remains wedded to the idea that the programmer is in control.
Accordingly, it's pretty provocative to suggest that a compiler or runtime would zero out memory without you specifically saying so. Any C programmer can overload malloc (doesn't zero memory) to calloc (which does). Whether it's a good or bad idea to do so is up to the programmer. The overall idea is don't do anything unless I say so.
No nukes get involved here - only core dumps.
So it's unspecified in terms of the standard, but specified by the implementation
This becomes a problem when a different implementation adds amazing tools for finding bugs (the sanitizer suite, for instance), and you can't use them because your code doesn't build with any other implementation.
When a different toolchain introduces a new feature for finding bugs that would be useful for Windows, the Microsoft compiler team can add that functionality to their own tools instead of porting Windows. An advantage of this is that they can customize the feature for exactly their use case. Yes, this is the definition of NIH syndrome, but that’s how large companies work.
This is probably way smaller than the whole Windows, but I would not be surprised if some MS dev are already internally compiling some of their components with clang for their own dev/testing (even if just for extra warnings, etc.)
And a major part of the work of the MSVC team today seems to be about standard compliance.
But yes, I do not really expect that they switch, and actually they probably don't even have the beginning of a serious reason to do so. This is not even a case of NIH. Their compiler derives from an ancient codebase and has been continuously maintained for several decades. They "invented" it. The only modern serious competition (that cares enough about Windows compat and some of their specific techs) has been started way after... They probably also have all kind of patents and whatnot about some security mitigations that are implemented by collaboration between the generated code and (the most modern versions of) low level parts of the Windows platform.
If you go deep and gnarly enough with your system, this is always the case, I'm afraid. There's a famous talk at Stanford by a Coverity... founder? consultant? about this.
This is always the case because the C standard is only a partial spec. There's no such thing as "a program written in C(++)", there's only "a program written in <Implementation> C(++)". If you compile your program with a different implementation, then it's a different program. It may work, but it may not.
What I'd really like to see is a modified version of the C/C++ standards which keeps the language the same "in spirit" but removes all undefined and implementation-dependent behaviour. This would give compiler writers a stationary (or at least slower-moving) target to aim for and make it possible for C to be portable in theory as well as (sorta) in practice.
Still, avoiding initializing padding is probably not a great example of a performance win through standards exploitation: in the example given it's not clear why you'd not just do one 8 byte zeroing write to cover the whole structure, rather than apparently splitting it into a 4-byte and a 1-byte write. Perhaps this was 32-bit code, where 8-byte write are slightly trickier, but even two 4-byte writes are likely to perform just as well or better. Probably it's just bad codegen to treat the initialization of the struct[11] in two parts: a single struct and a 10-member array.
The problem is the C code that relies on it: effectively you are using a dialect of C which gives stronger guarantees, so you lose the ability to use any other implementation which doesn't provide those guarantees.
How do you lose it in this case? You shouldn't been reading values from those padding bits anyway...
This can be problematic if you are copying kernel space memory to a user space process, for example. Let's say there's a call into the kernel that returns a copy of this 4+1 struct with three more bytes of padding. Maybe what was on the stack before the space was assigned to those last padding bytes are some information the kernel definitely shouldn't leak to user space, like some bytes of a password, and now any user space process could potentially read them simply by calling some unrelated kernel function.
This example also shows how "the semantics" is a fiddly concept. The reason the standard allows leaving bytes unzeroed is because they are not "semantically" important. But they actually do matter.
The problem with the mentality that it's always the programmer fault for not following "the rules" is they you eventually get to the point where the rules allow for no good solutions at all.
I believe C99 says chars and unsigned chars have no padding.
https://stackoverflow.com/questions/13929462/can-the-unsigne...
Language lawyers need not apply.
Who's relying on unspecified behavior? Do you mean some theoretical programmer? Well of course that's not a good thing, but I don't see what it has to do with the situation in the story and the MS engineers decideing to change how they implement that unspecified part of the spec.
An example of such an information leak is:
struct foo {
int a;
char b;
}
void send_foo_01() {
foo x {0, 1};
write(fd, &x, sizeof(foo));
}
Checking it after the fact is non-portable.
gcc, clang, icc all assume two's complement and happily overflow via -fwrapv. There's no such compiler on the Unisys anymore. They rather emulate two's complement. http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2218.htm
C++, but apparently something similar is being considered for C as well.
https://twitter.com/jfbastien/status/989242576598327296?lang...
I assert this 'it might not be two's-complement' excuse is nonsense.
I do agree with you, it's not a big deal for most devs. The only place I'd worry about it is systems where it might be expected to be in service for many years or intended to be ported frequently. In that case, you never know what sort of crazy architecture you may end up with in the future.
There's always something a bit icky with not sticking to the spec, but in the grand scheme of things relying on two's complement is not a big deal.
"Nowadays Unisys emulates their old architecture using x86 CPUs with attached FPGAs for customers who have legacy applications which they’ve been unable to migrate. These applications are unlikely to be well served by modern C++, signed integers are the least of their problem. Post-modern C++ should focus on serving its existing users well, and incoming users should be blissfully unaware of integer esoterica."
They are pretty open the future of systems programming on Windows is C++ and .NET Native.
For C devs, they have helped clang to work on Windows and there is WSL for UNIX like software.
Just like they have improved Visual Studio integration with clang and gcc.
As for Visual C++, the name says it all.
Their C love on the projects I mentioned, is via clang and gcc.
This "solution" also prevents code analysis tools from detecting reads from padding bytes as "non-initialized memory reads".
When a routine leaves sensitive data on the stack, the entire area used up by that sensitive data, must be wiped out before the routine returns (and still, wiping out padding bytes would not be required). How about the part of the stack space not covered by that array of 11 elements?
Otherwise, you'd be right: a shared stack would be a giant source of information leakage from kernel space to user-space unless it was very carefully managed (probably at a significant performance cost). Thus, separate stacks (it also has the advantage of not needing to make assumptions about how user-mode programs use their stack, e.g., if they are transiently using "unallocated" stack above rsp, etc).
Probably what happened here is that this structure was copied back to user space (e.g., as the result of a system call) exposing the kernel data.
But what on earth were they doing with the padding bits?
That would not have gone well with uninitialized padding :-)
More likely it was done to prevent Windows API calls from panicking when they accessed unset parameter structures.
As a former Windows programmer, that was one of the largest sources of errors back in the Win32 days.
If you write inline assembler, you can access this stuff anyway. So I'm not seeing what the value is in zeroing it by the caller. The kernel callee should zero its stack frame before returning.
Then, however, if this structure is copied back to user-space, e.g,. as an output parameter of a kernel call, the padding bytes with the exposed data will be copied along with it (unless you get lucky and the copy routine happens to make the exact same decision with regard to padding handling).
If the kernel stack _itself_ was visible to user-space, you'd have a whole separate set of problems: you'd have to zero the whole stack (or at least the extent of the stack that could have been touched) on every kernel call.
I recently learned of one such good engineering decision (I hope I'm remembering it correctly). Let's say you have a struct with an int32 and a byte in it. That's 5 bytes, right? But the platform alignment is a multiple of 4 bytes, so there's 3 bytes of padding (sizeof the struct is 8 bytes). If we stack-allocate an array of 11 of these and zero-initialize with = { 0 }, what would you expect to see in memory after initialization?
It turns out the answer was that the first element of the array would have its 5 bytes zeroed, but the 3 bytes of padding would be left uninitialized. Then, the remaining 10 elements of the array would be zeroed with a memset that actually zeroed all 80 remaining bytes. It sounds weird but this is a legal thing to do from the standard's perspective. All they're obligated to zero out are the non-padding bytes. This UB was leading to disclosure of little bits of kernel memory back into user mode because Windows engineers assumed that = { 0 } was the same as leaving the variable uninitialized and then memsetting the whole thing to zero. Nope!
The compiler team fixed this by always zeroing out padding too. Problem solved. There are some cases where it's not quite as fast. But it's the right engineering decision by the compiler team for their customers, both internal and external.