IANAL but you could likely successfully claim that you forgot where the key is, exactly because that's a product of the mind. If they have evidence that you do actually know then you might be compelled to hand it over, though.