It depends on the environment and who's listening to it. High end lossy is really good these days (great for casual listening), but I think most people could tell the difference between the two with a basic setup and a quiet room.

Of course, ymmv since audio is entirely subjective. If you enjoy listening to lossy music, you shouldn't let anyone take that away from you.