> because I can compromise the thing without opening it (just press the button and reflash with non-signed images).

I don't think so. With RDP enabled the only thing the bootloader lets you do is erase all flash. After that RDP is disabled and you can write your own image but at that point the keys are already gone.