News for hackers. Stuff that matters.
Re:C Programming Guidelines by jeffreyrogers on Tuesday September 01, 2015 @11:10PM (#10157220)
For memory allocation you often don't need to use malloc. Many (most?) programs can mmap a large region of memory and allocate from that. Then when you're done you just unmap the region. This is much easier to use since you only have to remember to unmap once as opposed to handling each allocated pointer individually.

Edit: Here is a paper that touches a bit on what I'm describing: http://people.cs.umass.edu/~emery/pubs/berger-oopsla2002.pdf

Also, to clarify, I'm not saying allocate yourself a big piece of memory and then reimplement malloc on top of that. You literally just get a big piece of memory and every time you want to allocate something new from it add the offset to the base address, return that to the user, and store the new offset.

Re:C Programming Guidelines by rc4algorithm on Tuesday September 01, 2015 @11:35PM (#10157289)
No no no no no. When you do this, you are:

1) Reinventing malloc for no good reason 2) Opening yourself up to a whole world of security holes

See Bob Beck's talk on the first 30 days of LibreSSL. Heartbleed wouldn't have been nearly as catastrophic if OpenSSL just used the system memory allocator like normal people.

Re:C Programming Guidelines by nitrogen on Wednesday September 02, 2015 @01:23AM (#10157571)
There are a few good reasons to reimplement malloc(), but of course it needs to be done very very carefully. For example, with a naive buddy list allocator it is (or was, in 2007) possible to more than triple allocation throughput vs. malloc(), but one loses many of malloc()'s security features.
Re:C Programming Guidelines by jeffreyrogers on Tuesday September 01, 2015 @11:51PM (#10157333)
I don't understand how this could lead to a security hole any more easily than using malloc could. Can you give an example? The system memory allocator is using mmap internally anyways (on Linux and anywhere else using the Doug Lea allocator at least) and a simple region based allocator is only a couple dozen lines of code.

The semantics are simple as well:

    region create_region(size_t size);
    void * allocate(region *r, size_t size);
    void destroy_region(region *r);
Edit: also, you're not reinventing malloc for no good reason. A region based allocator is much faster, possibly at the expense of using more space than the system allocator (because you might reserve more space than you actually need).
Re:C Programming Guidelines by adrusi on Wednesday September 02, 2015 @12:26AM (#10157421)
The point is that when you use malloc, an attacker exploiting a buffer overrun can't easily guess at the offset they need to find useful data. When you allocate everything in a big region, your process can be expected to read past the object bounds, so an attacker might be able to probe the address space (also the memory layout might be more deterministic). If they try to probe like that in a program that allocates with malloc, they'll just segfault the program.
Re:C Programming Guidelines by jeffreyrogers on Wednesday September 02, 2015 @12:47AM (#10157484)
Yes, this is a reasonable objection.
Re:C Programming Guidelines by SamReidHughes on Wednesday September 02, 2015 @12:23AM (#10157415)
If you want to count examples you're reasoning about this the wrong way. The question is what way leads to a higher probability of making a mistake, not the existence of discrete examples. Considering that one way, Valgrind works out of the box, but with your way, it doesn't, I think the answer to that question is quite clear. Even if there weren't tools like that, it is easier to read code and understand pointers' lifetimes when they're being handled individually, instead of having their lifetimes be part of some far-off region.
Re:C Programming Guidelines by jeffreyrogers on Wednesday September 02, 2015 @12:39AM (#10157458)
The whole point of using regions is not to be worrying about pointer lifetimes. And you don't need Valgrind when you have regions since you aren't going to leak memory by forgetting to free something. Valgrind solves a problem that doesn't exist when you use regions.

If you're really worried that you'll leak an entire region worth of data just allocate the region with malloc instead of mmap and then use Valgrind to tell you what regions you aren't destroying.

Re:C Programming Guidelines by SamReidHughes on Wednesday September 02, 2015 @01:19AM (#10157560)
Valgrind isn't for memory leaks, its main purpose is for catching out-of-bounds and uninitialized accesses.
Re:C Programming Guidelines by blt on Wednesday September 02, 2015 @12:46AM (#10157480)
The risk is that you'll screw up and write a buggy allocator with a security hole.

Yes, if your program just needs to allocate lots of memory, do some computations, then exit, this approach works. But the programs where security is most critical do not usually follow that pattern.

Re:C Programming Guidelines by jeffreyrogers on Wednesday September 02, 2015 @12:56AM (#10157503)
The code for this is so simple it would be hard to screw up. Plus you can just use obstacks[1] which pretty much provide the interface I was describing.

Look, I'm totally willing to accept that there might be flaws with this approach, but with the exception of adrusi, no one's objections have been all that reasonable. If there are legitimate objections (not just handwavy, oh, you'll probably implement the allocator wrong) I'd love to hear them. I use this pattern in my own code and I'll stop if there are legitimate flaws.

Also, Akamai released a patch for the OpenSSL allocator bug mentioned earlier. Guess what the patch used: mmaped regions.[2]

[1]: http://www.gnu.org/software/libc/manual/html_node/Obstacks.h...

[2]: http://thread.gmane.org/gmane.comp.encryption.openssl.user/5...

Re:C Programming Guidelines by z92 on Wednesday September 02, 2015 @02:48AM (#10157765)
Very good tip. Thank you. It looks like you are being opposed without a reason. Not everyone is writing encryption software in C.
Re:C Programming Guidelines by Sphax on Wednesday September 02, 2015 @04:05AM (#10157977)
Would you happen to have some code online I can look at which use this technique ? I first saw Casey Muratori from https://handmadehero.org/ use an approach like this and used the same approach in a small project of mine. It worked well and was simple enough to implement.
Re:C Programming Guidelines by setpatchaddress on Wednesday September 02, 2015 @12:27AM (#10157423)
Because you're going to have bugs. You don't need the extra performance, and you do need the extra safety. There is no good reason for you to do this in 2015, unless your system's malloc is broken. The exceptions to this rule know who they are.
Re:C Programming Guidelines by jeffreyrogers on Wednesday September 02, 2015 @12:35AM (#10157447)
Yes, safety was the initial motivation. The advantage is that you free once and the semantics are more stack like. The performance is an additional benefit.

It is entirely possible to have no bugs in the <100 lines needed for this code, so I don't think the bug argument is valid. adrusi did raise a legitimate concern however.

Re:C Programming Guidelines by caf on Wednesday September 02, 2015 @12:02AM (#10157361)
Setting aside the wisdom of this approach, if you were going to do this why wouldn't you just use malloc() to allocate the single large arena that you're going to allocate from yourself? It's still just one free().

In other words, the malloc-versus-mmap distinction here isn't the important one, it's the custom allocator versus system allocator one.

Re:C Programming Guidelines by jeffreyrogers on Wednesday September 02, 2015 @12:08AM (#10157376)
That would work too. malloc() uses mmap() internally for large allocations, so the reason to use mmap() is just to skip all the other stuff malloc() does (which you don't care about if you have a custom allocator).
Re:C Programming Guidelines by userbinator on Wednesday September 02, 2015 @01:24AM (#10157573)
For memory allocation you often don't need to use malloc.

True.

Many (most?) programs can mmap a large region of memory and allocate from that.

You don't even need to do that for many applications. Coming from an embedded systems background where dynamic allocation is highly discouraged and almost never necessary, the amount of superfluous allocation/deallocation I see in a lot of code is astounding; maybe it's because of my experience, but I can often very easily find a way to do something without dynamic allocation and consequently O(n) space, with what usually turns out to be simpler (thus less chance of being buggy) code and O(1) space.

But when you do need dynamic allocation, malloc() isn't bad at all.

Re:C Programming Guidelines by jeffreyrogers on Wednesday September 02, 2015 @09:48AM (#10159082)
This is a very good point.
Re:C Programming Guidelines by btrask on Thursday September 03, 2015 @09:16PM (#10168365)
Hey Jeff, I had to double check because I couldn't believe this was you. I'm a big fan of the Jeff and Casey Show!

The problem with doing custom allocation is not specifically due to bugs in the allocator per se, but because your custom allocator will probably not be hardened against other bugs like some other allocators are. Specifically you will piss off the OpenBSD people who have put a lot of work into their allocator with "exploit mitigations."

A bump allocator is pretty much guaranteed to let overflows (reads and writes) touch other valid memory. A hardened allocator will use guard pages and other tricks to prevent those bugs from being exploitable.

One thing you can do is use bump allocators within "security zones." For example data related to a single user can be allocated together without much risk, but data from separate users is kept separate.

This is very different from the world of game programming. I've been wondering lately if there will be some sort of bifurcation of the field.

Re:C Programming Guidelines by jeffreyrogers on Monday September 07, 2015 @04:44PM (#10182966)
Ahhh, this makes a lot of sense. Also, I think you're mixing me up with Jeff Roberts. I'm Jeff Rogers :). But yes, the Jeff and Casey Show is awesome :).
Re:C Programming Guidelines by btrask on Saturday September 12, 2015 @10:55PM (#10210193)
Oh darn!