True enough, there is other "noise" on the phone, in the form of point-like finger prints, and even other trails. But you're imagining a blank phone, where you have to try and discern one trail from another. Now turn the phone on, and the unlock background appears. Which trails intersect all of the dots of the unlock background?

It's much easier that you imagine. I've been using my phone as I normally do throughout the day, and I can see the unlock pattern clearly on the phone.

That's why you need a unlock pattern that intersects / overlaps itself. (Not foolproof, just makes that attack harder.)

It also doesn't work nearly as well if you are OCD about wiping your screen often, not that that makes the pattern lock any better but it isn't quite that trivial to defeat