Once the address is given out, it is the client's for the term of the lease.
Obviously, this is not realistic, and no one would ever implement a DHCP server that allowed this to happen: it is going to keep track of some fixed number of leases (probably/hopefully as many IP addresses as it is managing) and, when it runs out, it will be forced to reclaim an address from another user.
If anything, the fact that such a mechanism is not specified is simply going to make this "worse", as the specific implementation of how and when the reclamation happens is going to end up as "whatever makes sense to the guy working on that specific DHCP server implementation", and may constitute seemingly random behavior.
Ignoring the client is the only automatic mechanism DHCP provides for countering this.
You have to remember that DHCP was written back when it was assumed that a competent systems administrator managed the server and could manually respond to running out of available addresses (which was considered a likely sign of misconfiguration), and didn't attempt to specify what to do in the case where it happens.
Not giving out an address and notifying the systems administrator is what the DHCP spec recommends in this case. In fact, section 2.2 claims "The allocation mechanism (the collection of DHCP servers) guarantees not to reallocate that address within the requested time" so any reclamation of IP address on the part of the server is technically in violation of RFC 2131.
For example, for a highly-used access point, it would make sense to give out only very short leases - say, 10 minutes. That'd support a churn of 25 clients per minute if you're giving out addresses from a /24 (and there's no reason why you couldn't use a smaller netmask, down to /8). It'd also be sensible to ignore a single MAC address once it had more than a handful of outstanding leases.
It doesn't help with a malicious client - but then neither does breaking leases. The malicious client can cause enough address flapping to make the network completely unuseable. You really can't protect a wifi network against a malicious client trying to DoS it - there's a myriad ways to do that apart from DHCP.
The problem is the impact of the false "ip conflict" that is caused by this optimization. A device that detects an ip conflict should not have to guess whether it is an ip conflict caused by an optimization on another device. As pointed out in other comments an ip conflict can cause other systems to drop existing connections.
No one likes ip conflict notifications, or to have their connections dropped. Play nice. That's all.
NAT state tables won't have to stretch that far as they can time out unused connections just like they do already.
Would there be any unintended side-effects with doing this?
If you don't like the NAT, go ahead and issue IPv6 :-)
The usual (perhaps standard, I'm not sure) process has the device confirming its DHCP lease when coming back on, and in this situation the device would be notified that it can't have the old lease as it's been repurposed, and the server will provide another lease (perhaps after killing some yet another lease).
If this is an Apple device and it behaves in the manner described, after coming back "on" it will not consult DHCP, but rather it will reuse the lease it held previously. If the server has given that lease to another device, the Apple device will butt in, causing an IP conflict. Perversely, the Apple device will shortly discover this, actually do a DHCP request, and switch over with no indication to the user, leaving the other device to wonder why it had an IP conflict and how to handle it.
Is this incorrect in any way?