I was traumatized by my undergraduate calculus courses. And that was without the profs (who were all excellent) slipping in any "Unsolved Research problems." This entire thread has been like Trauma Trigger [http://en.wikipedia.org/wiki/Trauma_trigger]

I just want to make sure the undergraduate instructors reading this thread realize that doing this on the sly in a standard problem set would be a very, very bad idea. Not cool.

With that in mind, what do you think is the acceptable way to get students out of the answer-getting mentality?

I think what the professor did was exactly correct, have them experience something new mathematically, but give them the heads up that they weren't (likely) to be solving it, as it's a new pattern they hadn't recognized before.

I wonder how well things are likely to work if you give students explicit license to not try in an effort to keep them from feeling failure.

Not knowing is the important part. If you know the question is unsolved, you already have an 'answer' for it: it's 'unsolved.'