If the defendant wanted to argue that he no longer remembered the password (but would be willing to decrypt the drive if he could), then that's something that the judge would consider. But a judge can't (and won't) simply say "Oh, maybe he hasn't complied because he just forgot the password. I'm going to let him off"
Witnesses (in the sense of being called to that stand in a courtroom) are frequently asked to tell the court what they saw/heard/did, or from where/whom they received information. Refusing to answer may get them pulled up on contempt charges, and if they claim not to remember then the judge needs to decide whether they are lying.
Given the impossibility of proving (in an absolute sense) that the witness does in fact remember (at that exact moment), it's a game that witnesses are likely to get away with (hence the standard "I don't recall" answer from politicians and beuracrats), but it's not a universal solution to the "I don't want to tell you" scenario.
> At the contempt hearing, the Government presented several witnesses to support its prima facie case of contempt. Doe’s sister testified to the fact that, while in her presence, Doe accessed child pornography files on his Mac Pro computer by means of entering passwords from memory. Further, a detective who executed the original search warrant stated that Doe did not provide his password at the time because he wanted to prevent the police from accessing his computer. Doe never asserted an inability to remember the passwords at that time.