That effectively never happens, or hasn't happened for me. My relationships with managers have always been cordial but we both know the truth. As a developer, it is very hard to ever become anything but a developer. Companies say they have career paths for developers to follow that are non-management tracks, but they effectively don't. You...just...move...on. Any salary increases I have gotten were at the forefront of being hired on and my skill at negotiating was all make or break for me and I have screwed it up in the past.

I did just recently have a good turn where the recruiter who helped me land my current job helped negotiate a bit higher pay and a week of vacation. I wasn't going to push that hard because I needed to be working and not sitting on my ass, but he did this for me without my asking and it worked. I was pretty thrilled that after so many years, someone else gave a damn about me. First time.

The recruiter gives a damn about you because the higher compensation he negotiates for you, the more he gets paid (usually). It's in the recruiter's interest to get you the best deal possible, if he's acting as your agent. (It's the opposite if the recruiter works for the employer.) Of course, the recruiter has to walk a fine line because he doesn't want to be too aggressive and lose the deal, but they generally will work for you to a point. However, the flip side to this is that, working through a recruiter, you might not get paid as much because, to the employer, the total compensation is higher since they have to pay the recruiter's fee which is very large. It's kinda like selling something through consignment: they want the best price possible, since they get a bigger cut, but because they get a cut, you might be getting less in the end. But recruiters can get you access to jobs you might not easily have access to otherwise (because the company didn't post it yet, won't post it, didn't post it where you'll see it, etc.), so there's that factor too.

The reality of it is the recruiter is working for the employer. The recruiter just wants to fill the vacancy as fast as possible and get their paycheck. They don't care about what salary they place you at because they get much more money by finding good candidates quickly, and they are unlikely to ever work with you again. But they will probably work with the same businesses over and over. The recruiter getting paid more because you get a higher salary only aligns your interests weakly. It can still make sense to work with recruiters for all the parties involved, but you need to make sure you are negotiating on your own behalf and plan that the recruiter will negotiate more strongly for the employer.

Yep, this is akin to the Freakonomics "why don't real estate agents always push for the highest sales price" question. The gist being, it's a numbers game for them. The extra 5-10k they negotiate when selling your house makes a small difference in their commission, and usually comes at a disproportionate cost - either risk of the deal falling through, or time spent in negotiations that could have been better spent moving on to the next deal. It wouldn't surprise me if recruiters face a similar calculus (time spent negotiating on your behalf, risk of damaging their relationship with the company, etc.), and would definitely agree that at the end of it, you have to be responsible for your own outcomes, because no one else will be.

Employees have an inherent need for loyalty to work and face considerable risk just for following their true worth to new companies and situations.

This makes it impossible to adequately take employers up to your real market worth as a long term employee. Too many of your coworkers will accept 50% of their worth rather than risk stress and a crises if a change falls apart. So as long as the employer only needs a kernel of long-term employees only a union has the necessary leverage.

They also face considerable risk for sticking around when an employer has expressed interest in underpaying and overworking them or just outright firing them. If you're hiring people at 50% of their desired pay, then you're generally only going to get 50% of their skill. This is a double-edged sword. Employees also have an inherent need to accomplish things, and if you're not using them to their full capacity, that's also a risk of them leaving, or being completely checked out at work.

I think the value of unions and professional organizations in these discussions is something that gets too little attention.

One of the most important reasons software development is not like actual engineering is because the practices associated with actual engineering aren't valued in the world of software, even though the political components of employment (e.g. loyalty, longevity at a company) are considered signals of high(er) quality.

The vast majority of developers simply don't have quality employment options for most of them to consider long-term (> 3-4 years) employment with a single company to be a good economic or career decision.

I fully agree with you re. 'shake your hand' but I have to defend the line manager at large companies: too often, they have their hands tied with yearly performance evaluation processes and bonuses and will likely be unable to fetch the employee the necessary raise outside of working within aforesaid processes (i.e. getting the employee a promotion in a higher pay band, etc.), which takes time. I say this not to defend anyone but to try to present a more balanced picture.

In a smaller company, I would be worried about 'revealing my hand' to a potentially vengeful manager asking for a raise and I would rather just leave than try to get them to counter offer. That is, if I ask for a raise and s/he denies it, I would feel that I would probably be the first one to get cut when the time comes so I might as well leave. In a larger company, I might try to work with the manager on a 'career plan' assuming the manager in question is not toxic; the toxic ones will pretty much ensure that one gets a 'needs-improvement' type rating, which pretty much limits any movement for the unfortunate employee from that point on.