> People used to have to rack their own hardware, install PDUs, check temperature gauges, plug wires, configure the switches, setup RAID and backups, make sure you have that serial cable plugged in so you can telnet in in case the switches don’t work, and lock the data center doors without forgetting their keys back in the day.

Yes, and other people did this job, not programmers. People that loved tinkering with hardware, low-level OS stuff, networking, etc. We even had a name for them: "system administrators" and "network administrators".

Nowadays everyone's expected to tweak CSS and configure kubernetes in parallel.

We still exist. Our job titles just changed and more responsibility was put on our plates. Now we do the above work, plus manage automation (Jenkins pipelines, Ansible playbooks, etc) among other work. Not that the increased responsibility is an issue, just that the title makes us seem fancier than we are. I think that is part of the problem, everyone needs some ridiculous title now, a “lowly” sysadmin gets looked at sideways.

Yes. Although I worn that and other hats before and often together, I disliked being a sysadmin. I don't like being on call, it causes me to wake up at two in the morning to check a website. I do not like worrying about when to install a new patch.

So when my job title switched away from programmer to some kind of DevOps thing and I was shoved back into the sys admin role, I was unhappy about it.

That does depend in some areas of technical /systems programming back then you normally developers where expected to know the basics and be able to do all that - the original full stack developer OSI 1-7.

Trouble is they tended to be interesting but poorly paid jobs

> People used to have to rack their own hardware, install PDUs, check temperature gauges, plug wires, configure the switches

What you describe is what used to be considered sysadmin work. I too did quite a bit of it, but most of it went to people who specialized in those things. And no, I don't do any of it now. On any given day I'll probably deal with machines in at least a dozen geographic locations across two continents, and if I tried to enter any of them I'd be stopped at the door - no, at the gate before I even got to the buildings. Instead, I write code to interact with the deployment system and the provisioning system and the multiple monitoring systems and the automatic remediation system and so on. Is it better? It scales better, but as a matter of personal satisfaction and avoiding fury at other people's crappy code I think I'd rather be plugging in cables and configuring switches.

> I think because some of the hard stuff got a lot easier, the easier stuff got unnecessarily complex.

There's a lot of truth to that, and I don't think we necessarily disagree on the outcome. However we got here, whichever specific things make modern software development more unpleasant, it seems that it is more unpleasant than it used to be.

Re: the sysadmin work, app developers have to worry a whole lot less about it, oftentimes not at all. In startups, that's a huge plus. I still remember servers going down because some workloads were destroying the RAID arrays s.t. that they were drawing down too much power and overloading the PDUs in their racks. I ended up having to hardwire the distribution of jobs to other servers while we (actually it was me, I was at a startup) got the PDUs upgraded by talking to a sysadmin on the phone. (Amusingly at some point we had to fix some DNS while restarting one of the boxes so I had to tell the guy what to do by dictating vi commands.)

But yeah it's like over time we ended up scaling the crappiness of software along with the software itself. At the same time, I'm inclined to believe that software development at large is in a much better position to solve a whole bunch of other problems than ever before. That keeps me motivated personally, though I get the frustrations for sure.

I think it is funny you mention 10 years because I would agree with you, in the last 10 years we corrected a lot of the sins that we did in early web dev, but I think the lament reaches further back. Move back 25 years ago and it was much easier. A kid in his room could singlehandedly write the next big thing. Things in a lot of ways where much simpler then, it's hard to not look back on that period without a lot of nostalgia.

I was around 25 years ago pushing bits around SunOS boxes and when people were still figuring out CMSes. FTP and BBEdit ftw. Definitely some things were simpler, but stuff would fall over in a heartbeat. I still think of the day of 9/11 when major websites couldn't serve their homepages because they were bombarded by traffic.

Kids in their rooms still do write the next big things. The difference now is they now raise a ton of money along with it. They scale a whole lot further, and much faster. There are more entering the engineering profession through Lambda School and bootcamps and what not. They also leave around a lot of shitty code that usually ends up building some business value. Much of this was true back in the day. It's just gained a whole lot more momentum IMHO.