Now, there are several problems with this. First is that it's simply not as easy - what the others do has a real effect on your motivation. Your manager may not be also your employer, so he may not have good enough reason to appreciate you. Being too conscientious and good for a job may keep you in that job forever. Etcetera.
In the end, if you feel staying at this job prevents you from working as hard as you'd like, it's a valid reason to switch.
In a previous life, I had to ask a developer to pull up his socks and come in on time, for just the reasons you outline - I didn't want the people who were working their socks off to start feeling aggrieved (I was starting to pick up some sotto voce grumbles).
It's part of what management should do.
But yes, you are correct, it leaves those of us that do actual work feeling like things are unfair which is not a good environment.
Any tips for me? Just suck it up and try to do good at my job is what I've been doing for the last 3 years, but sometimes it gets frustrating.