For most people, it's pretty easy to cut 400-500 kCals out and barely notice.
Just cut back on soda. Eat half a burrito instead of guzzling the whole thing. Eat proper meals instead of snacking on candy bars.
Just the basics.
I've never dieted, but I'm down at 177 pounds from 189, since I thought I was getting a bit heavy, just through using the techniques above.
I suspect the largest contribution is from not eating an entire Chipotle burrito every couple of days, but instead eating just half, which is still a pretty good meal.
I totally agree with you on your last point about sloppy record keeping. You've learned over time how to properly estimate calories. Once I started to actually measure and record the food I consumed my weight changed dramatically.
On more than one occasion, I've made relatively mild cuts to my caloric intake (say, 250 Cal/day), and lost significant (>10 lbs) amounts of weight in 1-2 month time periods (and I've kept it off for years). Right now, I maintain my weight (6'2", 180lbs) by eating about 2,300 Calories per day. I don't count Calories anymore (I can estimate pretty well), but by weighing myself daily, I adjust what I eat to maintain.
People think that calorie restriction doesn't work, because they don't give it enough time, they aren't careful about it (i.e. sloppy record keeping), or they cheat. But if you're diligent, patient and honest, it's the most effective weight loss routine you can establish. I recommend using something like calorie-count.com to keep track of everything that you eat for a week or two, then cutting back on that number by a small amount (say, 250-500 Cal.) for a month. You'll be surprised at the results.