Sometimes the cowboy will linger around to try keep the unshippable code as-is. :( I do not understand why.
If you think you're documenting way too much then it's about right. The people picking up your source years after you've left the company will need that much documentation. Documentation also breeds more documentation. When someone goes in and changes something they're far more likely to document it if their change makes existing documentation/comments incorrect.
I see both code and comments as a three-way conversation between myself, the person reading the code (who may be future-me or someone else), and the compiler.
People who learn a second human language often fall back on a crutch known as "code switching". With code switching, when you don't know how to express something in the language you are learning, you switch back to your native language. This can often help people develop more fluency because they don't sit around concentrating on trying to find the correct term. They can say the parts that are difficult in their native language and then move on to the rest of the conversation.
However, code switching ultimately slows down your progress as a language learner. If you don't know how to express what you want with the correct grammar or vocabulary, it's usually better to try to find a different way to say it. By doing this you exercise your ability to be flexible in your speech. The code switcher tends to hit a ceiling very quickly and never becomes able to express themselves in the new language -- because they never have to. Finding alternate ways to express yourself is often frustrating, slow and embarrassing, but will help you in the long run.
Comments are like code switching. It's really helpful when you've written some code and you think, "I'm not sure that people will be able to understand what this means". It helps you get unstuck from your current position and to move on to the next thing, safe in the knowledge that you have expressed what you need to express.
However, you can also look at the situation differently. If you write comments, then you do not have to express yourself clearly in the programming language. It's easy to fall into the trap of thinking that it is unimportant to express yourself clearly because 1) you have comments (which are easy to write) 2) expressing yourself in code is hard 3) it will take a lot of time to learn how to do it well 4) you might fail and end up with difficult to read code.
These are all seemingly excellent reasons for writing comments, but they will also hold you back from being a better programmer IMHO. I prefer to avoid comments, using them as a technique of last resort. However, if someone says, "I don't understand your code" you have to take it pretty seriously and find a way to express yourself more clearly.
As a side note, this is one of those areas where reasonable people can differ. Of the issues that can cause conflict on a team, I find this to be one of the biggest. No matter what your personal opinion is, it's important to work as a team and do your best to please your teammates. Code is not for the writer -- it's for the reader.
There is often back-story that needs to be explained. Something as obvious as input sensitization may have a more specific meaning behind the specific implementation, for example a legacy Orcale DB client that chokes on certain special characters. A comment in a case like that can prevent you from getting woken up at 1am because a well meaning attempt to apply the same naming rules to schemas as you do to everything broke things.
If someone says they "don't understand your code" then either they suck at understanding code, you suck at writing it or both.
Comments are a tool to increase the speed and accuracy with which someone else can understand your code.
The code you're using to escape those specific characters should ideally be written in such a way that its intent is obvious even to a novice reading it. For the novice, then, the question shifts from "WTF does this do?" to "why is this here?", which would be a great example what to actually write comments for: context.
Imagine you're writing some very math-heavy library. Ideally, I shouldn't be a language expert to be able to see what your code does. This still doesn't help me if I don't understand the underlying math, however. Even hints like "this operation is called XY and can be used to Z" may be very helpful if someone wants to get a hold of what it does. With that, you can look up literature for this specific part.
I dont have a mentor, so this has helped tremendously.
But I agree entirely. I go back to things I did even a year ago and desperately wish I had documented more.
Now I'm left wondering if that makes a point for or against the OP's arguments. ;-)
This resonates with me, and I hope I'm better than that. I don't know if I am. Some metrics might suggest so, but who knows.
Over the last couple of years, I've been working with a repeating pattern. Teammates have a problem, but the problem can be split into 20% - 40% hard / algorithmic / architectural issues, and something between 60% and 120% other detail-work once the hard parts and a couple of easy parts as examples are done.
So, I've grown to tackle and solve the hard parts, and give people a small chunk of software so they can solve their problem on their own, as long as I help them a little along the way. This is powerful if you have a strong understanding of the problem to solve, and a couple of open-minded team-mates. And it helps everyone to grow on a technical level, making further problems easy.
However, this requires care, communication, a strong understanding of the issue, support for the solution... By now, people ask me for to handle the hard parts, and I can just dump them a repository with a README and a rough solution. But that was a long way, and giving less support, less care to do the right thing, and doing less to build trust with the team will lead to a catastrophe. And interestingly enough - oversupporting would also lead to problems.