>It seems like if these ever get challenged to the Supreme Court the current judges will rule with something like it being at the president's discretion.

Given that this is the same Supreme Court that ruled Biden (or Trump) could have them all shot[1], it seems near-certain that you're correct.

1. https://www.supremecourt.gov/opinions/23pdf/23-939_e2pg.pdf (JUSTICE SOTOMAYOR dissent, pages 29-30)