News for hackers. Stuff that matters.
Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by int_19h on Monday October 29, 2018 @02:08AM (#18325383)
I'm actually kinda sad that VLAs didn't work out in general (they have been downgraded to an optional feature in the latest C standards). For all the downsides, they make working with matrices in C much easier.
Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by enriquto on Monday October 29, 2018 @06:17AM (#18326245)
Yeah, VLA is one of my favorite features in C. I would love if it was viable to use them for arbitrarily large temporary arrays. They lead to much cleaner code. Instead of

    {
            float *x = malloc(n * sizeof*x);
            ...
            free(x);
    }
you do simply

    {
            float x[n];
            ...
    }

In image processing, you often need large temporary images, but it is dangerous to distribute code such as the above unless you play with the stack limits from outside your program.
Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by rwmj on Monday October 29, 2018 @09:08AM (#18327178)
It's non-standard (although we have tried to get it added to the next Cxx standard) but you can use __attribute__((cleanup)) here. In systemd code it is common to write this as:

    _cleanup_free_ float *x = malloc (...);
Since cleanups are supported by both GCC and Clang it's not a real problem to use them on Linux, BSD and MacOS. We recently added them to libvirt, have used them in libguestfs for a long time, and as I mention above they are used in systemd for years. They are also applicable to many other cases such as automatically closing files at the end of the current scope.
Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by dzdt on Monday October 29, 2018 @06:38AM (#18326338)
Its unfortunate that C and C++ kept the mindset that automatically memory managed variables are stored on a SMALL stack, with the consequence of exceeding that small and unknown size being that your program crashes.

For variable sized arrays, there is already a bit of overhead in sizing the allocation. Would it really have been impossible to move large allocations to the heap with automatic free at exit of the scope?

Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by exDM69 on Monday October 29, 2018 @09:41AM (#18327418)
> Its unfortunate that C and C++ kept the mindset that automatically memory managed variables are stored on a SMALL stack

This is not a property of the language, it's a property of the runtime its running on.

In a normal userspace program under Linux (at least), you can stack allocate megabytes and the stack will dynamically resize (note that shrinking the stack might not happen in a timely manner).

But in kernel space, there's a conscious decision to keep the stack small and not dynamic. While it's not impossible to have dynamic stack in kernel space, that has a lot of implications.

Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by carlmr on Monday October 29, 2018 @07:53AM (#18326703)
>For variable sized arrays, there is already a bit of overhead in sizing the allocation. Would it really have been impossible to move large allocations to the heap with automatic free at exit of the scope?

Not a compiler writer here so I'm not sure. I think it wouldn't be too difficult, but then this goes very much against the exact control you normally expect from C. And violating expectations like this is usually not good.

It would also produce bigger functions in a somewhat opaque manner, because you would need the code to deal with malloc and free and stack allocation at the same time.

Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by plasticchris on Monday October 29, 2018 @09:16AM (#18327230)
Not all software written in c has a heap.
Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by megous on Monday October 29, 2018 @11:47AM (#18328440)
Not all C programs have stack ether. Say Micochip's XC8 compiler for pic14 family.
Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by rbanffy on Monday October 29, 2018 @12:38PM (#18328857)
That has to be "fun" to work with...
Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by megous on Tuesday October 30, 2018 @01:17PM (#18338965)
You just can't recurse and the same function can't be called from isr and from main context at the same time. Basically, nothing is re-entrable. Otherwise it's quite fine. To be sure, you shouldn't be doing those things anyway if you have 64B of RAM and 512 words or ROM. :D
Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by cesarb on Monday October 29, 2018 @01:35PM (#18329357)
And even if it has a heap, the heap might not be available. In a kernel, the function might have been called in interrupt or atomic context; in userspace, the function might be part of the implementation of the heap itself, or a low-level function called before the heap is initialized.
Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by maxlybbert on Monday October 29, 2018 @07:46AM (#18326655)
C++ has std::vector, which doesn’t have any implicit size limitations. I haven’t kept up with the details, so this may have changed, but during the discussions for what C++11, the committee decided not to include VLAs in the official standard because the library already included std::vector.
Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by int_19h on Monday October 29, 2018 @01:46PM (#18329458)
The problem with vector is that it always heap-allocates. The ideal approach is the one where small allocations are on the stack, and large ones are on the heap, without the API user having to do anything about it.

C++ has had a proposal for std::dynarray floated for a while, which would basically have semantics allowing stack allocation, but wouldn't mandate it, leaving it as a quality of implementation issue (so it would be legal to implement it on top of vector, but it was assumed that compilers would go for optimizations given the opportunity). It didn't pass, unfortunately.

Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by maxlybbert on Tuesday October 30, 2018 @08:16AM (#18335776)
I guess in theory, you could pull it off with a very clever allocator. Bloomberg’s BDE has an allocator along those lines ( https://bloomberg.github.io/bde/classbdlma_1_1BufferedSequen... ), but I don’t think it made it into the standard (memory_resource header).

My point is only that runtime-sized arrays may have made it into the C standard, but they aren’t in the C++ standard.

Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by geezerjay on Tuesday October 30, 2018 @04:40AM (#18334782)
> The problem with vector is that it always heap-allocates.

How about std::array ?

Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by int_19h on Tuesday October 30, 2018 @04:44AM (#18334809)
It's not dynamically sized - the size is a template parameter, which means that it has to be a compile-time constant.
Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by kevin_thibedeau on Monday October 29, 2018 @10:24AM (#18327761)
It's unfortunate when system programmers write unportable code that depends on the assumption of having a large stack.
Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by enriquto on Monday October 29, 2018 @11:01AM (#18328036)
The word "large" does not really mean anything. You are always using the stack, and there is not portable way to check whether it is full or not. Now that is unfortunate.
Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by viraptor on Monday October 29, 2018 @04:15PM (#18330936)
There's still `alloca` which allocates on the stack and frees automatically.
Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by gameswithgo on Monday October 29, 2018 @07:34AM (#18326595)
seems you could macro this really easy?
Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by pjmlp on Monday October 29, 2018 @07:20AM (#18326509)
That is kind of what Ada does.
Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by int_19h on Monday October 29, 2018 @01:53PM (#18329529)
VLAs date back all the way to Algol 60. Most Algol-family languages inherited that trait, and many relied on it heavily. C (actually, B) excluded them because it was deliberately lower-level than was the norm for Algol family before.
Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by pjmlp on Monday October 29, 2018 @02:00PM (#18329593)
The big difference is that in the Algol 60 family bounds checking was enforced.

"Many years later we asked our customers whether they wished us to provide an option to switch off these checks in the interests of efficiency on production runs. Unanimously, they urged us not to--they already knew how frequently subscript errors occur on production runs where failure to detect them could be disastrous. I note with fear and horror that even in 1980, language designers and users have not learned this lesson. In any respectable branch of engineering, failure to observe such elementary precautions would have long been against the law."

-- C.A.Hoare on his Turing's award speech

C and B designers had another point of view regarding security, or lack of thereof.

Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by int_19h on Monday October 29, 2018 @03:23PM (#18330440)
That's orthogonal to arrays being variable-length or not, though...
Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by pjmlp on Monday October 29, 2018 @04:39PM (#18331170)
The point was that the way VLAs were designed in C99, it was a motorway to stack corruption, leading to dropping it out as optional feature in C11 and C++ not even considering them.
Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by int_19h on Tuesday October 30, 2018 @12:19AM (#18333990)
I don't see anything in the wording of C99 around VLAs that requires them to always be allocated on the stack. That implementations did so regardless is, arguably, a defect on their part, and a quality-of-implementation issue.
Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by pjmlp on Tuesday October 30, 2018 @03:59AM (#18334619)
Given C design culture how could you expect any other kind of approach to their implementation?
Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by int_19h on Tuesday October 30, 2018 @04:46AM (#18334823)
Well, the whole point of VLAs was supposed to placate the math/numeric crowd somewhat, and they generally expect things to be a little bit higher-level than what's otherwise mandated by the C "zero overhead" philosophy. Since VLAs are opt-in, code that cares more about perf could always skip them and not pay the tax.

But yes, it was probably too naive to expect that to work out.

Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by skocznymroczny on Monday October 29, 2018 @09:10AM (#18327191)
Hmm, how does it work if you want to pass that array somewhere else, and you want that other code to free it?
Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by lucozade on Monday October 29, 2018 @09:16AM (#18327229)
> how does it work if you want to pass that array somewhere else, and you want that other code to free it?

It wouldn't work, like a fixed size array wouldn't.

Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by pjmlp on Monday October 29, 2018 @07:19AM (#18326501)
It made it stack corruption way too easy, it was a very bad idea.

Languages like Ada do have them, but then again the language also protects against stack corruption.

Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by int_19h on Monday October 29, 2018 @01:48PM (#18329483)
There's no reason why VLAs have to be allocated uniformly on the stack. The C standard doesn't say that "auto" = stack. It just says that it has to be automatically deallocated when it's out of scope.

But even as implemented today, in practice, it all works fine if you're not working with really large arrays. In a lot of scenarios, you do not. This isn't really any different than using recursion when you reasonably assume that recursion depth is not going to be large. Sometimes it's just easier and clearer all around.

Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by pjmlp on Monday October 29, 2018 @01:52PM (#18329514)
That it the main reason why so many CVEs are related to memory corruption in C code as cause, in practice things do not work fine.

Which was the main theme at Linux Kernel Security Summit 2018.

Just for info, if not wanting to bother watching Google's talk, 68% of Linux kernel exploits were caused by out of bounds errors, including those caused by VLAs misuse.

Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by agumonkey on Monday October 29, 2018 @08:56AM (#18327081)
what do they do to avoid it ? static analysis ? runtime layout tricks ?
Re:The Linux Kernel Is Now VLA (Variable-Length Array) Free by pjmlp on Monday October 29, 2018 @10:56AM (#18327989)
With something that standard C keeps ignoring, bounds checking.

If the requested amount if bigger than the stack size defined at compilation time, an exception is thrown.

While C just overwrites the stack.