this is basically the xor canary approach originally pioneered by the Stackguard guys (i'm pretty sure you were already around at the time though probably forgot such old history as did the rest of the world apparently ;). the OpenBSD implementation suffers from a few problems, mostly their own making:

1. if they can't find a register to load the cookie into, they'll silently skip instrumentation (i'm not sure how that would happen in practice but the silent treatment when omitting a security feature is a non-starter).

2. if they can find such a register then it'll be spilled to the stack and restored in the epilogue, so a normal buffer overflow can control both the xor'd retaddr and the retaddr itself and the only thing standing in the way of exploitation is the secret cookie value - not unlike with Stackguard/SSP.

3. one would think that a per-function cookie is an improvement but... they're shared among threads (in userland) or everything (in the kernel) so infoleaks are just as catastrophic as before (it'd certainly help if someone described a proper threat model for this defense). at least the kernel side should use a per-syscall cookie to make it somewhat resemble an actual defense mechanism (and there's some more described in my presentation).

4. the int3 stuffing before retn must be someone's joke 'cos it sure as hell won't prevent abusing the retn as a gadget. it does introduce a mispredicted branch for every single function return however.

Hey PaXTeam, thanks for having a look! I wrote the implementation, so I can answer some of these.

1. We don't silently skip instrumentation. If we can't find a free register then we will force the frame setup code to the front of the function so we can get one. See the diff in PEI::calculateSaveRestoreBlocks().

2. We do spill the calculated value to the stack. This is unavoidable in many cases (non-leaf functions). It would be an optimization to not do this in leaf functions, but this would also mean finding a register that is unused throughout the function. This turns out to be a small number of functions, so we didn't pursue it for the initial implementation.

3. I'm not sure what you mean by the cookies are shared. Do you just mean that they are all in the openbsd.randomdata section? They have to live somewhere. Being able to read arbitrary memory in the openbsd.randomdata section would leak them, yes, though this doesn't seem to have been a problem for the existing stack canary, which lives in the same section. I see that RAP keeps the cookie on a register, which sounds like a neat idea. I'd be curious to see how you manage to rotate the cookie arbitrarily.

4. I'm glad you like the int3 stuffing. :-) We could always make the int3 sled longer if it turns out these rets are still accessible in gadgets that terminate on the return instruction. Have you found any?

Anyway, I'm happy to see your commentary on this. You guys do some nice work! If you have other suggestions for improvement I'd be happy to hear them. You can email me at mortimer@.

1. both insertReturnProtectorPrologue and insertReturnProtectorEpilogue check hasReturnProtectorTempRegister before proceeding with the instrumentation. so either the changes to calculateSaveRestoreBlocks are not enough to prevent that condition from ever triggering or these checks should be asserts at most or just be eliminated altogether.

2. sure but then this means that RETGUARD is not an improvement over Stackguard/SSP which is not how it's marketed...

3. shared means that entities of a class (threads in a process in userland, every single process/thread in the kernel) see the exact same cookies so leaking a cookie from one entity can allow exploitation by another. this is especially detrimental to the kernel side protection. frequent enough cookie rerandomization can help narrow this channel (RAP has a per-thread cookie in the kernel that is updated on each syscall, and there's some more to reduce infoleaks across kernel stacks, it's all in the presentation).

4. any normal path leading up to the ret is a gadget and int3 stuffing does nothing to prevent that (the underlying logic here is that if one can retarget a return to arbitary addresses then he has already leaked enough information so bypassing the cookie check is a no-brainer too). not only that but in the bsd.mp kernel i just checked, of the 32199 ret (0xc3) bytes only 20236 are actual retn insns, the rest are inside insns. so this int3 stuffing leaves many other instances available. Red Hat tried similar gadget elimination a while ago but noone's using the gcc feature as far as i can tell.

Thanks for the summary, comparison and criticism. Cheers

They put it into a system people will actually use.

Just leaving these here...

https://www.theregister.co.uk/2017/06/26/linus_torvalds_slam...

https://www.theregister.co.uk/2018/01/19/grsecurity_libel_ap...

This is just drama and isn't informative in any way.

It's informative in that it tells one that one should stay away from grsec's patch sets and why.

When the BDFL of one's kernel says something like that, combined with how radioactive the community interaction seems to have been in the past, the notion that one might get sufficient support or have positive interactions with the wider community while using the grsec kernel fork is dubious at best.

The idea that you should avoid grsecurity because Linus Torvalds, the "BDFL", says so is so messed up I actually don't know how to rebut it.

Either way: I'm not saying Retguard is based on RAP --- my question was, "what's the relationship between the techniques". But if there is a relationship, OpenBSD should be explicit about it, so that we can keep track of the evolution of memory corruption countermeasures.

I do not care whether you think people should or shouldn't run grsecurity.

I don't avoid grsec because Linus says so. I avoid it because he says they break things when they don't need to. Security is a trade-off and I trust Linus in judging that and in ensuring the operating system should never break user space, unless it absolutely has to. If it does, I sure don't want to be stuck going to a group that goes against the spirit of the GPL (by punishing users who redistribute by cutting them off from future access) and which sues prominent members of the community essentially for pointing this out; which pretty much ensures I will be stuck going to them for support and not redistributing their work as I am permitted to under the GPL. No thanks to that kind of coercion!

I too am curious what the technical basis is for the patch but your assertion that drama is irrelevant is dangerous when applied to community projects that exist because of the goodwill of their members. grsec should be called out, when mentioned, because of their demonstrated ability to make the code they do produce less than useful because of the encumbrance it carries due to its origin.

None of this has anything to do with my question and I'll ask that you not use my comments as a coat rack to hang your unrelated concerns about grsecurity off of.

I'm asking a research question, not a user question.

Thanks.

I was responding to your comment that the concerns raised by GP were drama and irrelevant. They are relevant to this subthread. I've said what I wish to say on the matter as well.

Oh, please. If you want to throw stones about behavior in the Linux kernel you're going to need to throw them at a hell of a lot more people than Grsec - there's decades of shit piled up.

Linus has always been a total blowhard when it comes to... everything, but in particular when it comes to security. I wouldn't take his opinions too seriously on the matter.

The fact is that grsec still maintains the state of the art for memory safety mitigations.

grsec may maintain state of the art, but it doesn't matter if one can't use their code because of the potential for breakage and toxic licensing conditions under which one would have to use it should one choose to do so. I also would not want to support a group which,according to the OSI, violates the spirit of the licence upon which their state of the art work is based - and which certainly violates generally accepted community norms.