The job right now seems just stitching AWS services together, and spending the rest of your time debugging and putting out fires.
The most uncomfortable truth of our field is that there is no floor for how bad code can be, yet still make people billions of dollars.
Because that's the outcome everyone else is seeking - making money. They don't care how good the code is. They care about whether it's making money or not.
The non-engineers also decided to write their own internal tool, with no review and zero attention to quality. It’s been outputting wrong results that will cost them a few million at least.
We eventually had to create a replica of Production specifically for such nonsense.
Well then we moved a lot more business logic that they "helped develop" that ran on a distributed platform closer to "real time" for the application, because it was very dynamic in nature, and couldn't be denormalized back into the DB and remain performant. So I had to build a new system for some of their ad hoc queries and teach them how to use it.
What eventually chuffed me was realizing the "R&D" person I was "training" on the system always wore a different three piece suit every day and at least in fashion signals was probably making a magnitude higher salary than me for slower, wronger answers than any system I worked on, but probably was great at throwing it (very slowly) into very fancy graphs in Excel and useless salesmanship.
There were a lot of reasons I already wasn't long for that job at that point, but that was a lasting lesson that quality and performance won't matter to certain styles of upper management.
What bothers me is that robustness and maintainability of code no longer seems to matter to engineers.
Believe me, the pressure right now to deliver working code no matter what is much less than, say, in early 2000's, when web development was very new and few people knew what they were doing.
just imagine if automobiles, aerospace or construction was run like that.... (hint, it was in the early days)
And then when a production bug is revealed, they then angrily ask "why are these errors occurring?"
Managers getting taken advantage of by allowing people to do anything in the name of 'technical debt' 'code quality', resulting in more bugs, lots of needless refactoring, tons of layers of abstraction and tests that actually do close to nothing.
Result is they are then surprised it takes a long time to develop new features where competitors do it faster and eat their lunch.
There needs to be some sort of balance, companies need to make money to allow for proper code to be developed, second is also valuable as long as it makes sense financially as well.
This heavily points to dysfunctional development team with fault being primary on tech leads making more and more mess the more they code.
> companies need to make money to allow for proper code to be developed, second is also valuable as long as it makes sense financially as well.
This one is merely about putting less resources on development. I don't think it fixes the dysfunctional team above, it is just avoid the issues. There is nothing that would make developers write useful tests. Nothing that would change tech leads, nothing that would prevent unmaintainable code appearing again.
Had the constantly refactoring team ended with very good code for too much of price, then cutting resources is likely to lead to better trade off. But that is not their case, their case is that they don't know how to write good code.
You are correct.
> I don't think it fixes the dysfunctional team above, it is just avoid the issues.
Sometimes avoiding the issues is good. My point is that some developers cry wolf too often, focusing them on solving problems and shipping features may yield better results.
Later "WE CAN"T PUT THIS IN FRONT OF A CUSTOMER"
X1 dev, X1 designer, e2 manager, etc. You can autoscale your company through jira integration.
You will save on health insurance, benefits, unproductive time, salaries etc. Only aws will need to cover that but those employees can work and scale for hundreds of companies monthly.
You can have your own college and production line for people out of school so you can grade.
Fortunately, this is a horrible idea that has essentially zero chance of becoming reality, as things like stable work relationships and a sense of purpose boost productivity, making employees more productive per unit labor cost than a Mechanical Turk for making more Mechanical Turks.
In the first half of the story, employees are increasingly micro-managed by a centralized software program which becomes more sophisticated and ubiquitous over time until humanity becomes fungible.
This is exactly what our engineers do, too! I am kept motivated by what we are able to do with what we've stitched together. But yes, knowing it is a bunch of plastic garbage underneath is... depressing, almost.
Maybe then go into the HPC/Defence/Air/Space/Masstransport-Industry, they will love you, no BS thats exacly what they want and need.
Bad metaphorical usage of the term "food chain" is an example of "office speak". Personally, I don't much like it.
I am interested in this topic, can you elaborate? Or any thoughts from anyone?
We just refuse to use vendor-lock-in cloud crap as a matter of principle. Regular server processes (though Haskell), regular machines (the NixOS), PostgreSQL, no problem.
I wonder if that's how the implementors of software that actually lasted for decades would describe their work or motivation.
Does this describe Unix, for example? In particular the old AT&T version, not GNU (or even modern BSD) code.
"The reliability of the basic utilities from GNU and Linux were noticeably better than those of the commercial systems."
[0] ftp://ftp.cs.wisc.edu/paradyn/technical_papers/fuzz.pdf
[1] ftp://ftp.cs.wisc.edu/paradyn/technical_papers/fuzz-revisited.pdf
https://medium.com/programming-philosophy/eric-raymond-s-17-...
At least to me. Having all these composable commands, unimaginable through how many iterations all that stuff went. I also think that Go was created in that spirit, with robustness/conciseness being more important than anything. Although it seems the language and its philosophy is adapting to more common ways of doing things.
in a group of skilled people it can still happen
is it due to bad leadership ?
You probably should get out of the HN bubble more often, a badly organized workplace is the norm not the exception.
(I hope it doesn't come as condescending since it's not my intention)
My comment is not about the norm but about the nature of the work structure. See I've mostly been working in average (non IT) places. I can understand average salesman and clerks having troubles organizing, dealing with computer issues, bad software, and cracking under managers because they don't have Masters degrees in abstract algebra applied to relational databases. So I assumed that in an IT company the skills would influence the structure. I guess it boils down to low level politics as usual.
May I interest you in one of my earlier comment to show you how dysfunctional a company can be (to put things in perspective : my boss was a software engineer before becoming the boss, our company has something like 25 millions € per year and the certificate should cost a few hundred euros) : https://news.ycombinator.com/item?id=24220464
Dysfunctional indeed.
#revolution
A little sensitive on this due to a recent experience where I was hired and given "ownership" only to have my very modest proposals for changes shot down if not just entirely ignored. Some "ownership"!
"No, there's no time for that, MVP, get something out! We'll fix it later!"
Truth is nobody cares for customer...
In software industry, cost of bug is mostly nothing compared to other industries like construction. Products are not always life-crucial. One person with laptop can build you product. In this context, engineering itself is secondary. Decisions are made on hype, not on reasoning.
But no, obviously not. Obviously it's bad engineering then.
Our company is looking for an experienced dev with Node and AWS experience. What's that, you have 15 years of industry experience but it's all on-prem Java/PHP? Too bad. NEXT!
Years later I worked at this place as a 3rd party contractor. What little bit of Spring I needed to know I picked up in the first few days. Their codebase and methodology was very poor and they were drowning because they only had one or two experienced people... but all the people they had had "Spring" on their resumes.
Open source is the only place I regularly see high quality code. There the devs are allowed to love their code like pets not cattle.
If you have nothing to sell you’re done. If your legacy cruft becomes overwhelming you’re done, but likely via a slow death.
Good teams are always struggling with this balance and there is no way to be right, but hopefully you can find that balance and be good enough.
I get that “done is better than perfect” but it feels like we’ve swung so hard in that direction that we seem to be largely neglecting stepping back and building the rough roll for the job in all but the largest places who have the luxury of having enough resources to assign a big enough team in the background to do it.
Safety stuff? Done then right way with the proper attention.
Rest of the production line stuff? Hack on hacks in order to get things online ASAP. At least that's my impression from the outside view I've seen.
The company who chooses to let the bosses son, who built custom PCs in high school, design the critical process control systems, is I'd say, outside the scope of the argument here.
As for "and then some" that's the part that's on senior management and HR, for setting up review/promotion structures that actively incentivize sloppy engineering. When all of the rewards are for New Shiny, some people will actively sabotage efforts to spend time any other way. Nobody wants to delay their new-feature effort to deal with new test failures or cleaned-up APIs. Even those who have enough of a conscience to to the right thing get kneecapped at every turn.
Rapid iteration is driven by customers not actually knowing what they need. There are places where "slow and steady" development would work, but very often you'd just find out what the customer needs isn't what they've asked for over a much longer period. You'd end up throwing out a lot of the perfectly crafted artisanal code anyway just as you would in the rapid dev version. The main difference is that the customer would have run out of money by then, and failed, and you'd need a new job.
Disregard for technical debt is certainly a force.
All these MBAs and non MBAs and Safe and Agile managers, they are all talk but dont allot time for issues that helps in better code-quality.
The problem is that this works only for large clients and conservative problem domains (such as accounting) where things change at a glacial pace. Back in the day, only such clients could afford enterprise/custom software, so using a waterfall methodology to develop that software worked well enough. Took five years to deliver the final product? No problem because the set of requirements you collected five years ago and based your software on probably still apply at the time of customer sign-off.
Things have changed. Software has gotten much cheaper to develop, which means it has become way more affordable, and the types of businesses who need that kind of software are in heavy competition with each other and their problems and needs change at breakneck speed. So your best best is to have your own software development practices to match that speed.
So just remove them. Now buildings have gotten much cheaper to build, which means they have become way more affordable, and the types of businesses who need that kind of building are in heavy competition with each other and their problems and needs change at breakneck speed.
Failing that, keep the regulations, and apply more to software. The "needs" of business will change, and competition will move. "Slow-and-steady" needn't be glacial, and there's a lot between that and "breakneck speed".
Plus there is a difference between time it takes to deliver, and frequency of requirement collection - your example implies you get requirements/sign-off exactly once before final delivery (5 years later). If we contrast "final" delivery with intermediate releases - why are the requirements not re-evaluated at each release time? You can still have slow, steady development, with lots of feedback - there just has to be an understanding that software delivered long term should not be interfered with short term concerns.
It might not be your ideal environment to be making products but it is our job to square the circle otherwise we’re not going to have jobs. For a concrete example look at the way game engine development happens. By and large engine development is slow and steady. Whilst making a game is something that massively benefits from blazing iteration speeds. Having that split between technology and product means you can do both.
Of course we’re also dealing with messy reality so things are faster or slower than each team would like at times.
Which is in turn driven by a venture capital industry that cares only about massive growth and nothing else.
But we're rewriting it to use Snowflake and Kafka. With no real use case for the rewrite.
The proven, you talk about, and the tech-debt.
If you mean well written c or Java code, that is well tested and clean. Then I'd prefer to then a modern JavaScript application without tests.
Legacy to me means applications that has turned into a mess with no tests. This can be a recent application or old.
The first thing is that I think people who don't want to work on legacy systems are making a pretty rational system based on the state of the industry these days. We recruit people heavily based on whatever they've done most recently, and for a lot of people even a short stint on a legacy system can be a career ending dead-end. In very competitive markets it might not be quite as bad, but for a large swath of the market (for instance in the Midwestern US, where I'm at), a single job with a legacy technology can make you nigh unemployable doing anything else. The problem is compounded by the fact that a lot of "recent legacy" technologies simply don't pay as well as either very legacy technologies (where you might be expected to have 20+ years of experience) or hotter new technologies.
Next, I think that we need to consider that, frankly, a lot of legacy code and technology sucks. New tools might have the same, or more, underlying fundamental issues, but in a lot of cases newer technology has a thicker layer of ergonomics on top, and the technologies themselves haven't ripened enough for the code smells to be noticeable, so from a day-to-day job enjoyment standpoint that somewhat lower paying job that is going to pidgeon hole you into the same boring BigCo work for the rest of your career is also going to come with a side dish of increased frustration. There's also a particular place in hell for certain types of legacy systems that really embraced the fads of their time- e.g. inheritance astronaut OOP hellscape Java applications from the late 90's, or rube-goldbergian metaprogramming funhouse ruby projects from the 2010's (I suspect the particular flavor of grotesque FP-look-alike cargo cult nonsense non-FP languages are trying to shoehorn onto poorly typed highly mutable environments is the fad that will give next decade's legacy system maintainers night sweats, but it's always hard to tell when you're in the middle of it).
Finally, and I think this is the biggest factor of all, a lot of companies with big legacy systems are digging their own graves because it's not just that they have legacy systems that need to be maintained, but they are so traumatized by the difficulty of keeping those systems up and running that they have calcified beyond any ability to experiment and try anything better, so it's not just that there are legacy systems to be maintained, all new work ends up getting shoehorned into the same legacy languages, legacy frameworks, and legacy business processes, because the fact that they never managed to migrate off their 1970's COBOL applications means that everything written today has to be designed to last 40 years using 20 year old technology. If engineers who were expected to maintain the legacy systems had some leeway to also experiment with newer technologies, and to modernize some parts of the system (with the acknowledgement that not all experiments would be a success), then it might be a lot easier to keep those engineers around.
The answer might be, because it got so difficult maintain, it was no longer worth it. And if the guy with experience of the system found it no longer worth it, you with little are unlikely to find it worth it. Beware the system abandoned by the long term maintainer, and then maintained after that by a string of short-term new hires. At best, the corp keeps increasing pay for every new hire in the hope they stay.
> had some leeway to also experiment
The idea behind the original "long-term professional" was a little autonomy for this kind of thing. Sadly, most corps refuse to classify developers as anything more important than cost centres, commodity resources, etc. Increasingly, they are contract hires, or easily-replaced temps (Accenture etc) and stuff like Agile abstractions, non-silo-ing, overstrict business time-management / task "sign-off"; and the trend of offering new hires better terms during negotiation, than improved terms to existing hires to keep them happy thereby encouraging no more than 2-3 years at any company to best improve your salary.
And it's not just young people. In the past I worked at a University where most teachers were unable (or refused to) to use the 30-year-old COBOL system. Those were people from late-twenties to late-sixty year olds, and they all hated it.
We needed extra staff just to type grades in the computer, since it was a long and onerous process, and that extra staff was very hard to retain.
The solution was redoing it in web and after that every teacher happily started doing it by themselves.
I work on aircraft design. We have people supporting legacy aircraft that were designed up to 50 years ago. I (and a lot of engineers) actively avoid those positions, and try to stick to programs that are less than 10-20 years old.
It's old technology, you don't get investment to improve anything, you don't get to work with modern tools (go back to working with scanned drawings instead of CAD data for example)…