I'm not sure what you mean by "valid." I think people can and should be kinder about asking, but it's always fair to ask "how did you learn that" if it's unclear.

But saying you don't remember is a fair response, too.