The talent does exist and is easy to find.

Problems these companies have are:

1. They are not really willing to compensate what it takes to attract the talent they really need. Most of these places start screaming bloody murder if you ask if they cover interview expenses or relocation, and trying to get them to actually state the range (which is no big deal to state if they really pay top rate since that's a huge recruiting advantage if true), is like pulling teeth. Which tells you they are not really paying top rate.

2. They aren't willing to do the work needed to recruit. This can include pushing hard from inside to make their company less unpleasant to work for, improving its reputation in industry, removing absurd contract requirements, treating workers fairly so Googling their name doesn't reveal the place is a nightmare and management are clueless. A lot of companies treat the employees as the enemy rather than an asset and in every case word gets out about this and good people stay away. A lot of companies are beseiged by politics so only the highly socially manipulative and dishonest do well. We see their representatives comment on boards and interviews on how it's impossible to find anyone, but going from their comment to a description of open positions that includes a clear statement of offered compensation is impossible. Companies are not reaching out to actual competent people and making offers. Instead they are hoping running an ad through a recruiter who posts ads on monster that often don't say what company it is, where it is, how much they pay, or clearly what it is they do. Why would anyone competent respond to these ads? Sure the desperate and incompetent and those that need visa sponsorship will respond, what about the people they really are trying to attract? Are qualified people really randomly searching monster these days? (Answer: no.)

3. Places with lots of mediocre and incompetent workers already in place simply can't attract talent at any price other than by the hour consultants who know they can leave at the end of the contract.

That's very true. We do actually do that with our production team. It's split into "ops", which is the day-to-day work and "engineering", which is the more complicated, more interesting work.

We hire junior people we think are smart into the ops roles and train them to eventually shift over into engineering. Some people can do it in a year or less, some a lot longer, and some just don't work out at all (or actually like doing the more routine work). Those people are definitely easier to find. We just talk to a lot of people and pick the ones we think are the smartest and will mesh the best and train them up on the rest.

In some cases, though, you just really need to get experienced people and can't wait the amount of time it would take to train.

What kind of experience are you looking for? Is it possible you are being too selective or misjudging the ability of experienced applicants?

That was definitely part of it, but we've changed our strategy a bit in an attempt to compensate. At first we were looking for people who had a lot of HPC-specific Linux experience. This, however, is a relatively small field because the cost-to-entry barriers are enormous if you're looking to do it at any great scale. That definitely restricts the set of potential candidates to a small subset of the set of "Linux people" (people from gov't labs, research universities, very large corporations, etc.)

Since we're willing to pay relocation expenses and buy people out of non-competes, we were doing a nation-wide search, but as a generalization, university and gov't employees are usually there because they like the relaxed environment and are willing to sacrifice some pay for it. (I have worked at both, so that generalization is drawn from only my experiences and those of my friends.)

So after getting few bites there, we rewrote the job description to just be very good with Linux in general, hoping to get a really smart person with skills that complement ours and then being able to train them in the HPC-specific stuff. That increased the number of applicants, but also dramatically decreased the quality-applicant to applicant ratio.

You're definitely right that I could be misjudging the ability of some of the applicants, but a great majority can't pass the Linux equivalent of FizzBuzz, even though they list 10+ years of experience.

What would be the Linux equivalent of FizzBuzz?

I'm thinking of it as a test for the simplest possible thing I can ask someone that will tell me immediately if they're worth talking to or not. I mentioned this in another post, the one I use that people bomb most often is: "I want to run this command on 100 files. Write me a for-loop in the shell of your choice to do it."

That's kind of terrifying that someone can't do that can say they are an SA. Did anyone you interviewed mention xargs as a BTW solution after presenting the for-do loop? I'm currently trying to figure out if one of the SA's at a current client actually does not know how to use tar, gzip, and scp, so I know where you are coming from, but still.

When you previously said "complicated applications in (python|perl|ruby)", did you mean someone who knew both front- and back-end? That implies someone who also knows Javascript, CSS, HTML, and a few databases, not to speak of one or more web frameworks, proxies, load balancers, etc.

When you say "debug and port C/C++ programs", did you mean someone who knows enough about Windows to port a program written there to run on Linux? Or someone who knows enough about BSD to port to Linux? Or someone who knows enough about [AIX|HP-UX|Solaris] to port to Linux? Port what, pure userland programs or stuff that messes around with ioctl calls? Debug from a hand-built in-memory circular log, core with no symbol table, using Dtrace, or just some judicious printfs or logging?

Depending upon how you specify what you want, you don't have to go far these days to stray a long distance from a "Linux SA" role, where even a "Senior Linux SA" job title won't cut it; you are moving closer to a Free Electron, and those people don't come cheap.

"Sorry, we don't have time for training."

"Sorry, we can't afford to send you to a week of training for $3000, you have to get this stuff done right now."

"Sorry, we're on a spending freeze."

They will pay whatever it takes* to get IT talent that does actually exist, at other companies.