News for hackers. Stuff that matters.
Re:Reading the NSA’s codebase: LemonGraph review by isaachier on Friday August 10, 2018 @10:07AM (#17733364)
To be fair, the way malloc is used is actually the preferred way. If you change the type of the pointer variable, the malloc is still valid.
Re:Reading the NSA’s codebase: LemonGraph review by drewg123 on Friday August 10, 2018 @11:17AM (#17734056)
I agree. I always do it this way for this exact reason.

However, I'm surprised the author did not point out that they are calling malloc() without checking for a NULL return value.

Re:Reading the NSA’s codebase: LemonGraph review by jimrandomh on Friday August 10, 2018 @12:41PM (#17734830)
Checking the return of malloc() for NULL is not currently considered good practice, because (a) on most platforms malloc is guaranteed not to return NULL, even if the system is out of memory, and (b) on the few platforms where malloc can return NULL, handling that case in practice basically never works.
Re:Reading the NSA’s codebase: LemonGraph review by loxias on Friday August 10, 2018 @08:09PM (#17737559)
> Checking the return of malloc() for NULL is not currently considered good practice

Why?

To quote malloc(3):

  On error, these functions return NULL.  NULL may also be returned by a
  successful call to malloc() with a size of zero, or by a successful call to
  calloc() with nmemb or size equal to zero.
I'm not disagreeing with your assertion that on linux, in practice, it doesn't return NULL -- (in fact, I didn't know that! and would appreciate being pointed at whatever I need to RTFM...) -- I just prefer code that's conservative, that follows the "letter of the law" when it comes to return values.
Re:Reading the NSA’s codebase: LemonGraph review by kazinator on Friday August 10, 2018 @08:17PM (#17737586)
If you write code that tests for null out of malloc, you have to commit to that code as a feature and have a test strategy for it. Simply porting the program to a platform where malloc returns null isn't enough; there are various cases to probe, like your very first malloc failing because the program was started in a low-memory environment. Or the case that the 99th malloc fails after 98 good ones.
Re:Reading the NSA’s codebase: LemonGraph review by loxias on Friday August 10, 2018 @08:31PM (#17737625)
Oh, I'm not saying you should test for NULL and then, uh, not care about the result!

If you port to some new platform, of course you need to go back and check the docs for new error codes.

For instance, I generally abort() on NULL after printing a message.

Re:Reading the NSA’s codebase: LemonGraph review by msbarnett on Friday August 10, 2018 @08:42PM (#17737654)
> For instance, I generally abort() on NULL after printing a message.

Ok, but...how are you printing the message?

On many platforms, with many standard libraries (glibc, for instance), printf et al will themselves malloc memory, so you're testing for a system telling you it couldn't allocate memory and then in response, potentially triggering further quite-likely-failing allocations that may or may not be handled by the library.

This is why people generally don't bother to check for NULL mallocs. Even if you're on a system that will even tell you you're in that situation (rare, these days, outside of the embedded world), there's almost nothing useful you can do.

Re:Reading the NSA’s codebase: LemonGraph review by convivialdingo on Friday August 10, 2018 @09:40PM (#17737835)
You’re absolutely correct, in this manner of development you just hit a wall and call it a day.

A few years back I read quite a few papers on NASA’s development standards and did quite a bit of kernel dev also.

I no longer do any allocations on daemonized or driver code like this. It’s far better to preallocate pools, store temp objects on stack and generally never malloc outside of the pool management.

When we hit the alloc wall, we now have well-defined behaviors and can gracefully choose whether to reject, clean or error out. Additionally our code is generally more performant, less buggy, and easier to debug.

Re:Reading the NSA’s codebase: LemonGraph review by loxias on Friday August 10, 2018 @11:02PM (#17738040)
You're absolutely correct. (full stop)

Honestly, in my own code I try to avoid lots of little heap allocations (duh), and when writing embedded things I don't have printf or malloc anyway.

Re:Reading the NSA’s codebase: LemonGraph review by kazinator on Friday August 10, 2018 @09:38PM (#17737828)
"care about the result" means:

- come up with a requirement about handling the bad result

- put in the code which does it

- test the code

> I generally abort() on NULL after printing a message.

On most modern systems, accessing the null pointer will also abort; you just won't get the nice error message. It will look the same as if the program hit a runaway recursion, or corrupt memory. The location in the program where it happened is equally traceable either way (with a core dump or debugger). Neither a segfault nor abort() do any graceful cleanup of anything. Basically, this is only a tiny increment in software quality over not handling the NULL at all.

Re:Reading the NSA’s codebase: LemonGraph review by PeCaN on Friday August 10, 2018 @07:48PM (#17737497)
>on most platforms malloc is guaranteed not to return NULL

This is only true on Linux, and isn't even guaranteed there. I'm pretty sure the default overcommit handling mode is still to refuse allocations that are much larger than there is free memory to support. (This was the case around 2.6.)

It's generally preferable for software to check if malloc() failed and have a chance to recover.

Re:Reading the NSA’s codebase: LemonGraph review by tjoff on Friday August 10, 2018 @01:14PM (#17735105)
Malloc will in some cases return null when the allocation is too large. This might happen long before system is out of memory and it is very much indeed preferred and practical to abort allocations in those cases.
Re:Reading the NSA’s codebase: LemonGraph review by anyfoo on Friday August 10, 2018 @01:50PM (#17735374)
Those are rather specific cases, though, aren't they? I'd imagine malloc only returns NULL when virtual address space is exhausted, not physical memory. So if you make an exceedingly large allocation for a very specific, massive thing, then yes, you might want to handle that. But if your allocation was rather small, then you are back to what your parent commentator said and are probably better off aborting.
Re:Reading the NSA’s codebase: LemonGraph review by drewg123 on Friday August 10, 2018 @03:08PM (#17735982)
It depends on whether the system you're using overcommits memory. Some OSes (eg Linux) provide settings to limit virtual memory use to some percentage of physical memory. See http://engineering.pivotal.io/post/virtual_memory_settings_i...

The only time you should avoid checking malloc() return values are for special mallocs that really cannot return NULL. One example is FreeBSD's kernel malloc() when called with the M_WAITOK flag.

Re:Reading the NSA’s codebase: LemonGraph review by tjoff on Friday August 10, 2018 @06:43PM (#17737223)
On a 32bit process that is easily achieved and also if the virtual memory has been fragmented.

I've encountered many situations where malloc will return null in real world applications. Some of which we really wanted to handle that scenario and fail gracefully or continue with less memory usage (and thus less performance).

Re:Reading the NSA’s codebase: LemonGraph review by kieckerjan on Friday August 10, 2018 @02:02PM (#17735491)
Eh? When did that get started? You are claiming resources. Not checking if you got them is a leap of faith, period. If nothing else, you can abort the program before nastier things happen.
Re:Reading the NSA’s codebase: LemonGraph review by flafla2 on Friday August 10, 2018 @07:10PM (#17737342)
This is correct. What happens when you are trying to rely on malloc'd memory for sensitive calculations? Reading an invalid pointer is undefined behavior. I see no reason why "it almost never works" to check for NULL and abort if necessary.

> Checking the return of malloc() for NULL is not currently considered good practice

This is simply false. I was a TA for the intro C class at CMU last year, and we still teach to check for NULL. In fact, we mark down students who do not do so.

Re:Reading the NSA’s codebase: LemonGraph review by Chabs on Friday August 10, 2018 @10:31AM (#17733541)
I'm mildly surprised to see the NSA using malloc instead of calloc, especially for allocating what seems to be a partially initialized struct.
Re:Reading the NSA’s codebase: LemonGraph review by isaachier on Friday August 10, 2018 @11:47AM (#17734392)
I understand the question, but as @AndyKelley taught me (https://github.com/ziglang/zig/pull/993#commitcomment-289183...), it is actually worse to zero a value unnecessarily if you can use Valgrind/sanitizers to check for uninitialized values. Initializing the value as zero will prevent the detection of a bad value.
Re:Reading the NSA’s codebase: LemonGraph review by foobiekr on Friday August 10, 2018 @10:58AM (#17733823)
Really depends on whether they know that all fields will be written before return or not.

This way hey are calling malloc() is very common in professional code bases and a safety measure; I’m surprised the author was surprised.

Re:Reading the NSA’s codebase: LemonGraph review by mediocrejoker on Friday August 10, 2018 @12:04PM (#17734546)
The way I read it, the author is surprised that struct node_t is typedef'd to node_t* and not simply node_t

I am not used to that style so it seems odd to me, but I suppose if it was common in a codebase one would get used to it.

Re:Reading the NSA’s codebase: LemonGraph review by convivialdingo on Friday August 10, 2018 @09:51PM (#17737861)
Most compilers in the 80’s and 90s had pretty restricted grammars based on K&R before C89 and newer standards fixed these sort of things. I’ve seen a bit of code like this from then... Off the top of my head, I’m thinking the Amiga toolkit and Xwindows. Old embedded code may still have some of this going on as well.
Re:Reading the NSA’s codebase: LemonGraph review by foobiekr on Friday August 10, 2018 @10:52PM (#17738005)
Sometimes limits are good. They lead to people playing it safe.

I've spent most of my life dealing with C code dating from 1986 and earlier to present day. Once you get used to the stylistic conventions one of the things that falls out is the simplicity even for huge, old legacy code bases. Often little more than grep or etags is sufficient to competently navigate and follow the flow of this kind of code. Simple things are simple: what callstacks can end up here? etc. The code often has the property that if you printed it you'd still be able to navigate it without issue. Given a random page and a line, getting to somewhere would be possible.

In comparison so many modern code bases are just completely incomprehensible without a tool in the form of an IDE (and often still quite challenging with them because even the IDE isn't sure, and so you must resort to runtime analysis). So many "tricks" used which require tools and require the tools to be bug free and robust.

The closest I have come to "pick it up and you can read and understand it without assistive technologies" is Go.

Re:Reading the NSA’s codebase: LemonGraph review by Matthias247 on Saturday August 11, 2018 @12:42AM (#17738394)
Passing sizeof(value) to malloc is really fine. But I consider the typedef of a pointer to a node is to the same identifier as the node itself to be pretty bad.

It's confusing for readers and new team members. And can even lead to strange when someone forgets to type "struct".

Re:Reading the NSA’s codebase: LemonGraph review by EpicEng on Friday August 10, 2018 @11:52AM (#17734428)
Why would that be? You don't cast the result of malloc in C. It's wholly unnecessary, adds clutter to the code, and potentially hides an error if you forgot to include stdlib.h.

So why would that be "the preferred way"? I've been writing C for some time and no one who knows C casts the return value of malloc.

Re:Reading the NSA’s codebase: LemonGraph review by optymizer on Friday August 10, 2018 @12:02PM (#17734523)
The parent comment was referring to the usage of sizeof().

Also, the fact that they cast the result of malloc makes me think they were using a C++ compiler, which will complain that there's no cast.

Re:Reading the NSA’s codebase: LemonGraph review by EpicEng on Friday August 10, 2018 @01:10PM (#17735065)
>The parent comment was referring to the usage of sizeof().

Ah ok, makes more sense in re-reading now. I was lazy and went off of a child comment.

>Also, the fact that they cast the result of malloc makes me think they were using a C++ compiler, which will complain that there's no cast.

Definitely could be that.

Re:Reading the NSA’s codebase: LemonGraph review by blattimwind on Friday August 10, 2018 @12:18PM (#17734658)
If you're compiling for C++, you should be using static_cast<> instead.
Re:Reading the NSA’s codebase: LemonGraph review by pksadiq on Friday August 10, 2018 @10:51AM (#17733741)
> To be fair, the way malloc is used is actually the preferred way.

Is it really right? In C, isn't it considered bad to cast malloc result as it masks some mistakes? Also, the code doesn't check for NULL for the malloc result.

Re:Reading the NSA’s codebase: LemonGraph review by isaachier on Friday August 10, 2018 @11:51AM (#17734415)
I am not referring to the lack of null check. Just the fact that it uses `malloc(sizeof(*x))` instead of using `malloc(sizeof(Type))`.
Re:Reading the NSA’s codebase: LemonGraph review by kevin_thibedeau on Friday August 10, 2018 @11:59AM (#17734502)
The latter creates Heisenbugs if you change the type of x and don't track down every malloc to update the type. It is better to avoid using sizeof(typename) if possible.

Now the profligate use of leading underscores. That is an issue.

Re:Reading the NSA’s codebase: LemonGraph review by saagarjha on Friday August 10, 2018 @03:52PM (#17736264)
> Now the profligate use of leading underscores. That is an issue.

This is a pretty common way of shoehorning access control into languages that do support them natively.

Re:Reading the NSA’s codebase: LemonGraph review by kieckerjan on Friday August 10, 2018 @01:51PM (#17735386)
Good point about the type change risk. However, in this case that was obviously not the reason, since there is an explicit cast (although the compiler would prevent it from becoming a HB). His hand was forced by the pointer hiding in the typedef. As I see it, this code is half-way smart, which is worse than no-way smart. (KISS!)
Re:Reading the NSA’s codebase: LemonGraph review by posterboy on Friday August 10, 2018 @01:48PM (#17735350)
where's the difference if you put in the numbers by hand? Which numbers do you put in to avoid mentioning the size of the type?
Re:Reading the NSA’s codebase: LemonGraph review by kevin_thibedeau on Friday August 10, 2018 @03:03PM (#17735934)
You just feed sizeof the instance of the object you're allocating memory for. That entails funky pseudo-dereferences for pointers so the compiler doesn't give you the size of a pointer. It will look up the object's type and requisite size to fill in the argument to malloc(). If the object in question has an unknown type (maybe all you've got is a void *) then you will have to provide the size by other means.
Re:Reading the NSA’s codebase: LemonGraph review by orbifold on Friday August 10, 2018 @11:09AM (#17733960)
Malloc does not fail under Linux unless you are in very peculiar circumstances.
Re:Reading the NSA’s codebase: LemonGraph review by krylon on Friday August 10, 2018 @11:29AM (#17734198)
Not everyone is running Linux.

And anyway, it is dirty - like crossing the street without looking for cars because traffic around here is very mellow, or peeing without washing your hands afterwards.

(That being said, I once helped somebody to write a program whose task it was to deliberately get the host system to the point where malloc(3) failed. It was not as trivial as one might think.)

Re:Reading the NSA’s codebase: LemonGraph review by jcranmer on Friday August 10, 2018 @11:24AM (#17734131)
Malloc will still fail for overly large allocation sizes. If you try to malloc(-1), for example.