If they’re using some special hash algorithm that retains most of the information of the original password and allows them to determine that “password1” and “password2” are similar and “blahblah7” is not, their approach is hopelessly broken too.
Requiring a plaintext password storage somewhere is an instant regulatory fail in ISO 27002 and PCI DSS standard. You can technically store the passwords encrypted, but attacker is liable to steal your salt/key, and protecting the passwords in transit strictly requires strong PKI. We know users cannot determine that anyway and get phished/MITMed.
Plus it's kind of mean. If your users are determined to ignore your policy, maybe it's a bad policy or you need to tell them to stop doing that.
Checking for x latest passwords requires just storing their hashes.
No, it’s not. It’s called locality-sensitive hashing. You’d store the last 10 LSH hashes in a database next to the SHA-256 hashes. Compare those LSH hashes to the LSH hash of the new password. Match? Reject the new password.
It’s also used to identify photos that are similar:
Well yeah. Protecting against this is the exact reason passwords aren't stored in plaintext.
Mind, I think the feature is misguided, but it is probably possible to implement it safely.
Same way you verify the current password: hash it and compare to what you have on file. If you use salted hashes and the salt changes, you'll have to keep enough of those around, too.