I've talked with a lot of different Indian and similar 'offshore' engineers, developers and managers and most of them ended up in the place they did due to choices between being a farmer or being an IT person, being in a poor province vs. being in a less-poor province, and being able to support your parents. Almost none of them had an upbringing, background or base passion for technology or IT in general.

While you can in theory do your IT things with no passion at all, and kill all creativity and still be productive, it often ends up being just 'mass' like you describe, a transformed version of the classical western 'consultants' that usually end up being companies that focus on sales and billable hours and never really deliver what the people needed (referring to earlier HN posts about accenture and deloitte).

I get that at some point all companies converge on the same generic point of being in the business of making money at all costs, and making it 'measurable' via sales, hours and contracts. It doesn't matter if it's software, financial, legal etc. At least not to them at that point.

The bigger problem with this is that the IT systems are not as well defined or a closed system as the other examples might be. A ledger would be simple to verify in comparison: you apply a fixed set of rules to a fixed period of data and it must result in something within a known range. With software you can do a little bit of determining that the 'borders' of your 'box' of functionality would be, but to do an exhaustive verification is so far from profitable that it's only done in a few highly regulated systems like flight controllers.

So now we have a system where a lot of money and power vectors are involved that would (somewhat) work for legal and finance and the likes (i.e. the classical big ones that existed long before software did) and a constant attempt to apply it to IT and hide everything they don't like about getting the wrong results. It still eats resources like the other kinds of work, but doesn't produce the results you would expect for the resources that went in to it.

Software doesn't operate in a vacuum, disconnecting users, buyers, sellers, or stakeholders in general is bad for everyone and because of endless change and interaction the whole model of 'buying a product' hasn't been realistic since 2000, and with connected OT (think SCADA and the likes) it won't even apply to embedded systems anymore.

"I've talked with a lot of different Indian and similar 'offshore' engineers, developers and managers and most of them ended up in the place they did due to choices between being a farmer or being an IT person, being in a poor province vs. being in a less-poor province, and being able to support your parents. Almost none of them had an upbringing, background, or base passion for technology or IT in general."

This sadly happened to me many times. Since the quality of many Indian teams was extremely abysmal, I found out that many so-called developers had no IT background at all, as you wrote. They went through a 2 to 4 weeks HTML/CSS/JS Bootcamp, and, very important allegedly in India, received a certificate, that qualified them as “Middle Web Developers”.

if one is looking for $10-20 hourly developer, certainly the result will likely be what you've experienced.

Big IT Indian firms usually will choose one or two highly talented developers for a project team and others just hop around them. Usually these projects aren't also very engineering-intensive and becomes boring once the base framework is created for creative folks. Good devs don't stay longer in a company at least during initial years simply they're in good demand and gets double the package with every jump. Eventually they'll make it to Indian subsidiaries of US companies like Microsoft/Google or, they'll start their own expert consultancy/product companies.

For the record I've worked out of India exclusively and have charged up to about $100 hourly rate from my US clients. I've gone through above process - my last job being at Microsoft India (2011-2016)

When developing nations that are primarily known for their cheap products start creating competitive high quality products what happens is that they start charging just as much as every other developed nation. That same dynamic plays out with Indian software developers.

If they can deliver X amount of value they can also charge X for their service. This means if you hire an Indian for $10 an hour you're going to get value equivalent to $10.

Since productivity per member drops as you increase team size you can't just hire a bigger team to get yourself out of this problem. The true number of developers that your project needs is probably constant and quite low.

If you hire cheaper but less experienced developers then it might simply take 6 times longer to finish the project than if you had hired an Indian developer that charges $60 per hour. This is usually where the project fails. Managers expect the project to be done in 6 months but this estimate is based on competent developers with 3 years of experience, not beginners who are working on their first project and first need to gather 3 years of experience to adequately finish the project.

Really the only way you can get a real bargain is by hiring someone, letting them gain experience without giving them a raise. The difference between market rate and actual developer salary is where you save the most money. The tricky part is creating a environment in which people want to stay.

India has quite a way to go to break that boundary. The IT industry in India is basically a giant demonstration of the Dead Sea effect. The most talented engineers from India have been leaving the country in droves for decades. The local industry has a deservedly terrible reputation, which motivates more local talent to leave, further degrading the local labor market, which further incentivises people to leave...

I think the most promising signs for the local industry is that it’s starting to produce some rather successful home grown startups. Which are much more likely to attract local talent than the outsourcing body shops are.

You get what you pay for. That holds true everywhere.

Umm... 20$ an hour, meaning 3200$ a month, would be quite a nice salary in a lot of Europe (Greece, Bulgaria, Romania, Poland, ...). You get people with BAs or MAs in comp science who have been developers for 2-5 years.

While I'm sure some will say that 3.2k is ridiculous in country X, I personally met people in all of the above countries working for less while being more experienced.

20 USD/hour is not what developer earns, but what you pay for developer time. Overhead (buildings, managers, computers, toilet paper), sales commissions, taxes, and off time and eats big big part of that money.

I've seen rate cards for quite a few bpo providers and major it body shops, and unless it's like an extreme case (Chennai and one or two other cities), the devs are getting slightly less than 10% of their billed rate, the rest being overhead.

Hell, I look at the delta between what my firm bills me out for and what I take home, and it's about 12% of my fully loaded day rate.

I've seen two type of compensation schemes during my agency years: ~60% take home, but zero during my downtime and around ~15% when I was salaried developer, which make sense as utilization of around 60% was expected.

The rule of thumb used to be, total costs = what you pay them * 1.5 to cover that stuff.

So 3200 becomes $4800 (or just under $57k a year) - that's still a good deal given what you'd get.

I've worked with eastern and central European developers and some of them are phenomenal (the skill breakdown is about equivalent to UK devs as a pool maybe tending towards them been slightly better but there will be some element of selection bias in that, language barrier is the biggest hurdle, many of them speak/write good English - my partner is Hungarian and her written English is at least as good as mine) but when the tiniest miss-understanding on requirements is costly during development and potentially devastating in production that really starts to matter.

I think you are misunderstanding what the $20 refer to. They are what the customer is paying to the company for a single billable hour. That money is not only used to pay the developer but usually the rest of the company too. Sales, marketing, accountants and other employees that aren't directly assigned to a project or are waiting for their next project will have to be paid from that billable hour as well. You can call them greedy or whatever but usually they get away with a 50% cut or sometimes more. Ok, now you are left with $10 which represents the labor cost + office space + equipment needed by the developer himself. So the employee gets paid $6.66 per hour before taxes but the customer is paying $20 for that.

Indeed. The starting salaries of fresh grads in (Wipro/Infosys/TCS/etc.) has been stagnant at INR 300,000 to INR 350,000 / year for the last 10-15 years. Converted to USD this is: ~ 4000 - 4700 / year.

20 USD / hour * 8 h/day * 21 days/month * 12 months = ~ 40,000 USD / year.

Of course, experienced ones don't receive that might higher salary either. About 5 years of experience would probably get you around double to triple the starting salary.

I guess that is the reason why there is interest in outsourcing to Eastern Europe. Case in point being EPAM Systems. Indian IT consultancies might have huge operations but they are not the only show in town.

Yes, and we collaborated with a few dozen EPAM developers on some projects at my last full-time corporate job, and all of the ones I worked with (both the in-person EPAM teammates and the remote ones) were very sharp people who definitely enjoy technological endeavors. As development firms such as EPAM gain traction, I expect some of the massive Indian firms will lose ground.

Right. I'm counting on that to happen. This should force the hands of the Indian firms to innovate or perish. That should be very interesting to watch.

Eastern Europe is in a wierd time where it has the infrastructure, market access, and human resources of a rich country, but much lower cost of living and pay.

That isn't sustainable - we will just see things equalise with Germany and then where is the competitive advantage? So cutting costs by pushing technical work to eastern EU isn't sustainable, at least in the 10 year time frame.

I am sure EPAM and the tech industry there will adapt to the shift but they will not be as the new Infosys.

The developer you're getting from the big IT consultancy is likely getting only 15-20% of what you're paying for him.

Definitely! I'd love to earn $3200 a month here...

I don't mind much about the wage gap between countries when I do local work, but it gets ackward when working with remote teams.

Like I may earn $20k/yr here for what my teammates abroad would be making $100k/yr.

These are IT agencies though, competing in a global market. Not individual salaries.

> Almost none of them had an upbringing, background or base passion for technology or IT in general.

> The bigger problem with this is that the IT systems are not as well defined or a closed system

This is the big issue. At the end of the day creating truly good usable software is both an art and a science. Data structures, algos and theory, but also thinking outside the box, asking inquisitive questions, challenging product and design.

This doesn't lend itself easily to mass export. Also, culturally, I've found that the West has a different way of thinking about things. In India, questioning your superiors is not something that is done often, but in the West it's common to challenge your boss on ideas.

You used example industries of legal and financial and even in those, passion and innovation are force multipliers. The domain-specific systems in these industries are every bit as ill-defined and open as software, it just looks very, very different. As a layperson to these industries, most people are only exposed to the pedestrian mainstream aspects, kind of like being exposed to only the teenager manning the Geek Squad desk and bringing them a WinPC to defrag to "speed up".

Dig not that many layers deep, and you'll expose the ill-defined and open systems in those industries. This is true nearly everywhere in nearly any human endeavor. This is why the Japanese kodawari concept (meticulous care in craftsmanship, design and service) is even feasible across an entire culture for centuries and counting. Once you really start caring about the customer, your entire center of gravity in your relationship with them changes, and you start stepping through the process through their eyes to anticipate their needs and empathize with how they likely feel through the process you set up for them to go through.

This is very different from the Western "attention to detail" concept. Personally for me, the Western concept has less of a connotation with focus upon caring and empathizing to "listen for" the details, and more upon a situational awareness to "watch for" the legalistic steps of a spelled-out procedure (perhaps rooted in an Industrial Age paradigm?).

> The bigger problem with this is that the IT systems are not as well defined or a closed system as the other examples might be. A ledger would be simple to verify in comparison: you apply a fixed set of rules to a fixed period of data and it must result in something within a known range. With software you can do a little bit of determining that the 'borders' of your 'box' of functionality would be, but to do an exhaustive verification is so far from profitable that it's only done in a few highly regulated systems like flight controllers.

That's a really good point. We haven't even settled on how to weed out bad programmers through a formal structure yet, and we still sometimes hear about companies setting KPIs based on LoC or bugs fixed. I'm not sure if it's even possible to have an objective framework to evaluate an outsourced system apart from "meets the spec" which is usually not very helpful unless there is in-house expertise spending a mountain of effort building the spec in the first place.

As a former Accenture Consultant I do have to point out we rarely used hourly billing. Fixed fees were preferred and I think it actually works better as it forces everyone to keep talking about scope. That being said, the worse part of Accenture was working with the offshore developers. The only way to move a project forward was hand holding these guys. I ended up working around the clock to catch our Hydrabaud team at the beginning of their day and at the end of thier day. Keeping in mind I was suppsedly dealing with the "manager". Was Accenture perfect? No. But based on what I've seen, projects were well run with componetent local staff, good managers and the client side was usually in shambles.

> But that is not how software is done. I want outcomes not masses. I need creativity to work on tools expressed through software not cheaper developers.

Fully agree, but the people making the decisions at large corps are not developers, and are just as focused on costs as the Indian outsourcing firms pushing low cost devs. And in turn, those people are pushed by the board to make cost reductions, and board members understand this even less.

As a developer and architect, I've had this discussion many times over my years working in the enterprise space, and it always ended with me throwing my hands up and giving up.

Non technical folks under-appreciate the cost and frequency of long-term technical debt.

It's easy to save money on developers, yet very quickly produce a net-negative result for the company.

Sadly, few companies have guardrails to prevent these sort of shortsighted choices. E.g. requiring parallel approval of projects by IT architects (whose existence admittedly carries its own hazard).

I've heard non-technical folks with their heads in the clouds expecting to run millions of dollars in transactions through an IT system, but cringed at my suggestion that they're likely to dump more than a couple hundred grand one-time into the scheme. Yet the same person doesn't blink at paying significantly more than that yearly maintaining existing low-tech systems, and has already spent millions on ERP projects.

I think a lot of this effect is caused by wishful thinking from people embarking on business plans which they would realise are non-viable if they did the accounting properly. People are buying a fantasy instead of an IT system.

Well, we all know why that is the case: uncertainty. If I pay $100k for a thing, I'm actually paying $100k for a chance at the thing by the date. Most likely, I'm going to end up paying 2x for the thing by 2x the time. Happens everywhere.

And that's not because the guy making it is shit. It's because while he's making it, the market is moving and I'm watching it, and when it moves enough I have to come over to him and tell him to change things.

That's why slow, costly things overrun by more and are later. It's because the world changes, and fast+cheap things are superior because even if they're worse, they exist and can therefore be changed to what the world is.

The difference is that an existing IT system is real and can be purchased for X amount. A potential future one cannot - and while devs are able to estimate (poorly) how long it will take others cannot and there is a real human bias - in fact one could say the future system is the fantasy

Purchases are good in the short term. You can evaluate them and pick the most suitable. Once you have settled with one product and know what you want it may make sense to work on optimizing your business with a custom fit solution.

It's not just that non-technical people under-appreciate technical debt.

If you put in the time and pay the money to build an awesome system that is stable and usable for decades, there's only a small chance you'll be rewarded. First because no one notices when software is working. Second because you will probably have moved on to a different role at a different company after your wisdom is confirmed ten years later.

Saving money now is what gets attention.

> Non technical folks under-appreciate the cost and frequency of long-term technical debt.

Undoubtedly this is true, but there's an implication here in this context that somehow on shore, or more well paid developers don't create such technical debt, when in fact they do. It amazes me that I see team after team of competent well intentioned developers create one ball of mud after another. It's striking really.

I think software development misses a "doctrine". It is very hard to develop robust maintainable software while trying to chase the latest fad (edit: under constant pressure to deliver new functionality).

I have known this as a junior, never really getting to know a technology before hopping to the next one, and I am still having to deal with this now that I have a bit more experience, trying to convince my colleagues not to be so quick to dismiss the modern for the newer.

Sometimes I feel like I do not know what we are doing although being one of the senior members of the team. Stuff quickly snowballs into mud... It's like there is not enough effort or time to deliver functionality while also staving off entropy

> It amazes me that I see team after team of competent well intentioned developers create one ball of mud after another. It's striking really.

It is! But I don't understand... how? How does that happen? You should put good people together and get something good. But it is not always so. Very puzzling to me.

Oh, look, the answer is easy. Manage these projects with SAFE. SAFE is a corporate friendly Agile methodology that essentially says, "Fuck that agile stuff. Use the Agile vernacular to keep being as unproductive as you always were."

This is like heroin to these corporations. They use Jira and Rally to create a cult of numerology games that reduce tasks to filling time-boxed, random things- to-do. Be sure the Agile coach is drilled in the game and clueless about "technical-things". Add an equally clueless Product Owner whose background is Word and Excel and you'll create the best Big Ball 'O Mud anybody ever wished for.

And if you're running late, add a dozen or so new inexpensive -cough- "developers" from your favorite outsourcing firm.

That's literally the job of a CIO/CTO - manage technical capability of the company.

If you have a bad CIO/CTO - you're screwed long term.

I don't think that is how these companies see themselves. For them they are just a source of cheap labor and the only reason a company would engage with them is for their low cost. They think that the engineers and the talent is easily replaceable and they even think they do adequate work nothing earth shattering. So the company's only aim will be to out maneuver the other IT consulting companies by undercutting on price.

You're both describing a 'market for lemons'.[1]

[1] https://en.wikipedia.org/wiki/The_Market_for_Lemons

Thank you for that article. I like to think that I keep up with what some call "Freakonomics" (not the book, but the concept), and I had never come across that succinct and well developed model. Again - thank you.