Sure, it felt like absolute chaos some days, but I"ve never felt more productive and in a flow as then, mainly because we weren't chasing the latest and greatest thing but focusing on what's working for us and perfecting it.
It's going to happen to anyone after 10-15years in the industry.
A resume full of less-shiny but still relevant technologies, otoh, has a lot of potential. You have fewer places to look for jobs, but you're a lot more exciting to the places which do want your skills.
Over 30 years that’s allowed me to be employable across a lot of different tech stacks.
Here's how I stay sane -- I focus on the changes. For the technologies that you noted, here are some good questions that come to mind to set them in my brain:
- What's the difference between CGI scripts and Serverless?
- What's the difference between Terraform and Ansible?
- What's the difference between a Bash script and a Makefile ? Is one more general than the other? Does one have a considerably better interface than the other?
All the solutions look the same, but the distinctions between them (even at a relatively surface level) are what help to actually tell things apart and stop the feeling of endless churn.
I think of it this way -- unlike other fields because the fruits of our research are immediately usable and almost zero-cost, we're given a front seat to the frontier of experimentation. It's a water hose, and you have to determine which streams are worth turning on/off, bottling for later.
But a lot of it, I can admit, boils down to sheep shaving. When you start figuring out how you'd implement codebase 2009 in stack 2023, and then actually doing it, that's golf. It's all good, it's educational, but it's downright immoral to sell that to a client and drag them through it.
AKA Yak shaving[0], which I am... unashamedly fond of doing
>> There are yaks to shave all over the stack, and I go where the yaks are. (quote is from my github readme)
> When you start figuring out how you'd implement codebase 2009 in stack 2023, and then actually doing it, that's golf. It's all good, it's educational, but it's downright immoral to sell that to a client and drag them through it.
I definitely agree that selling this to a client when you've told them you'll do the minimum to update the codebase is immoral, but if there is value in updating, then you should expose them that value.
For example -- moving a client to AWS might not make sense, but if they're maintaining on-prem infra and paying tons of money for an application that would actually cost less as a single ECS service (or App Runner, these days)... Then maybe you should upgrade their stack a little bit.
In the end it's your job as a consultant or an employee (if you're high level enough) to introduce change that will make the company better, if you see an opportunity.
Tech stacks crumble underneath their own weight, and if they're bad enough, become business risks. If it's humming along, easy to maintain, and isn't riddled with security holes, then great!
I basically agree with you, if the cost of building the code will be easily outweighed by the savings to the client and I also think the platform will hold them for the next 10 years, I'll pitch it [e.g. I've moved many early clients from my servers over to AWS services when it made sense, and still maintain them]. But it's easy to see too how if I were trying to constantly drum up more work, it would be seductive to pitch them on new hyped up technologies they don't need, just to remain relevant and keep the checks flowing. I think that's the animating force behind all of this platform churn, not excited developers who just want to sheep or yak-shave (thanks. I always forget which ruminant I'm shaving ;)
> - What's the difference between Terraform and Ansible?
Marketing and VCs
But also, those are some pretty odd comparisons. For sure Ansible and Terraform aren't directly comparable. If anything they're complementary. Terraform provisions machines (and also infra, etc), and Ansible configures provisioned machines.
Definitely -- I'd say that these questions force the understanding of the fundamentals, because that's what you get down to ("fundamental"/irreconcilable differences).
The problem is that fundamentals can be similar, and sometimes the difference is elsewhere (UX/DX/perf for example).
> But also, those are some pretty odd comparisons.
So they are odd comparisons, in some sense -- but the point was to reflect the kind of questions someone who didn't know the answer would ask. Ansible and Terraform (and Salt/Puppet/Chef) are often presented in similar context and they're often confusing precisely because they are so close but different.
The other example (CGI vs Serverless) is situation that's somewhat more targeted towards the age discussion -- did we churn over the years for churnings' sake?
> For sure Ansible and Terraform aren't directly comparable. If anything they're complementary. Terraform provisions machines (and also infra, etc), and Ansible configures provisioned machines.
What you've said is the usual rebuttal (and it's mostly right!), but did you know that Ansible did/does provisioning?[0] The lines aren't drawn as neatly as they seem to be.
You can build a Terraform-like experience out of Ansible, if you wanted to, and it's important to know that and why you choose a tool like Terraform instead of traveling deep into Ansible land.
All that said, if you prefer, replace "Ansible" with "Pulumi" (Full Disclosure: I'm firmly on the Pulumi side in the Terraform vs Pulumi debate).
[0]: https://docs.ansible.com/ansible/latest/collections/index_mo...
I wrote everything in shell, then puppet, then chef, then ansible. Doing literally the same thing. Because fads.
Do you really not see any difference between Ansible and a shell script?
Even just the move between shell and bash/csh/tcsh/zsh whatever else can make a marked difference in productivity and ergonomics.
Surely they do, but how much those differences matter depends on where those tools sit in relation to what you're trying to achieve.
For example, I could write essays about the differences between bash and PowerShell, and about differences in productivity their various facets create, in both abstract and concrete terms. However, when my task is not about shell scripting per se, and the script plays only a minor role in the solution ("oh, but we can make the button launch a script that launches X"), and my main concern about that stage is to convince the team to use Bash and/or PWSH instead of making Python a dependency in the project - then bash and PWSH are really the same thing to me - any difference in productivity for that use case is dominated by one's familiarity with the tool, and none of it matters anyway if my co-worker succeeds slotting in Python for that use case.
Similarly, there are many differences between bare shell scripts and Ansible, and there are differences between Ansible and Puppet and Chef too. But they're also close to each other, so if your use case doesn't hinge on those differences, you can be excused for wondering why can't we just keep using a Makefile for this.
I'm still using it, though every piece of software lately, fad or not, is something I tend to endure or survive or cope with, rather than use or enjoy using. Little papercuts everywhere.
In the middle I wrote puppet and chef. All doing basically the same stuff. And yes, obviously there's worlds of differences. Using Puppet killed my productivity, for example.
But my point was that those tools are more comparible than 'terraform' vs 'ansible'. I stand by that comment.
See my response here[0] as well, but to summarize:
- There is more overlap than it seems on the surface (not implying that you are taking a surface view)
- I wanted to get across was the way someone might ask if they were new/looked at it all as churn from the outside. The average dev who thinks "devops churns too fast" is not necessarily going to know the difference between ansible and terraform to begin with, never mind knowing that ansible/salt/puppet/chef are a different approach/lineage compared to terraform.
> And yes, obviously there's worlds of differences. Using Puppet killed my productivity, for example.
I never used Puppet -- it was love at first sight with Ansible for me, felt like the perfect amount of abstraction/structure even though some of the patterns were long in the tooth.
My career is not as long as yours, but I still use ansible to this day (with pulumi).
In the same way that a toaster can be used as a hammer if you hit nails hard enough with it from the "correct" side. Ansible barely does it's main purpose, configuration management, but abusing it for provisioning is just on a different level. Even overlooking the extremely narrow feature set, just `state: absent` should be enough to convince anyone Ansible just isn't made for provisioning. Add it the bolted on templating of a configuration language that uses spaces for logic, the fun of Python dependencies of which you need a ton to do any provisioning, and you're just in for a massive world of pain for literally no reason.
Disclaimer: I work at HashiCorp, but not on or with Terraform, and have had a disdain for using YAML for anything other than basic configuration and Ansible for anything other than simple, basic, low complexity and low scale Configuration Management since I inherited an Ansible project to manage VMware configuration in a past job.
And use version control for the things I can ...
Each of such tools expands to take the role of the other (and also read email)...
Sure - you could use tf’s local/remote-exec to do the stuff you’d normally do with ansible but man that would be annoying and slow going when you know you can reach for ansible.
Vice versa, you could have an ansible playbook to call the cloud provider’s api and cook up some kind of approval workflow / state management but man that would be annoying and slow going when you know you can reach for terraform.
> you could have an ansible playbook to call the cloud provider’s api and cook up some kind of approval workflow / state management
Was generally what people did before Terraform, and they be reluctant to add another complex tool, significantly overlapping with the current complex tool, just to simplify a part that already works OK.
Conversely, if Terraform is really so amazing as one would believe reading HN, then some amount of "local/remoe-exec" pain seems justified if it lets you avoid bringing in Ansible with all its complexity?
All these tools cost not just storage and processing, but also brain space. A Makefile in a cron, running some shell scripts, may be more brittle than whatever cookbooks or playbooks or ${what's the name for Terraform's flavor of makefiles}, but is also might be easier to fix when something breaks.
Ansible can provision virtual machines as well, and from my experience it is better suited for it too, since it doesn't save state outside the system. So there's lesser chance for it to get out of sync.
It's mostly culture. People these days learn GKE or Kubernetes first, and their tutorials mention Terraform so that's what they'll use. It used to be managed VMs and that culture was around Puppet, Ansible or Salt.
State serves a purposes, it's not just there for fun. Ansible will lose track of your VM if you rename it (and can barely handle you renaming it with itself), and won't delete it if you just delete it from the configuration file.
The system has state. The question is how we want to manage it. The design in this case chooses to opimize to lessen the impact of errors. All real world systems are prone to errors in the long run.
If real world parables are excused: If the map (serialized state) and terrain (system state) differs, is it a good idea to insist on following the map?
> Ansible will lose track of your VM if you rename it
Sure, but so will Terraform. Or worse.
> won't delete it if you just delete it from the configuration
Sure it can. There are no limitations there. The only questions are whether you want it to, and how you want it to.
Ansible is a superset of Terraform. It's mostly a matter of culture which tool people choose, and which you think is the easiest to use.
If you're doing landscaping and the map is your requirements, yes. And that's what infrastructure is comparable to (hence Terraform's name btw)
> > Ansible will lose track of your VM if you rename it
> Sure, but so will Terraform. Or worse.
No it won't, because Terraform stores the underlying platform ID in it's state. It will just know there was a rename, and will inform you it plans on rolling it back to what you have requested in the configuration. It won't delete it. Ansible will want to recreate it, with all conflicts this could bring.
> Ansible is a superset of Terraform. It's mostly a matter of culture which tool people choose, and which you think is the easiest to use.
It's a different tool for a different job (at which it isn't great due to using a markup language which really doesn't lend itself to templating with lots of templating, and a dynamic language that suffers from version and dependency hell).
[1]: https://en.wikipedia.org/wiki/Narcissism_of_small_difference...
Now everything thing looks like a shell script or json exchange.
Ansible is a bunch of scripts. Terraform is a bunch of json exchanges.
There is maybe some progress on a academic level, but I don't think we progress fast on the fundamental level, on how to write better code. There are slight improvements here and there, but it is mostly just different. And more complex. Or rather new tools are needed because everything gets more complicated (some say bloated).
That being said, the new thing that I am learning is general purpose GPU compute. That is quite new and useful (but hard). But it is an improvement.
It's the same ideas, but I will assure you that most of the large things that people get crotchety about are legitimate moves forward.
People were complaining about "docker" (i.e. cgroups + namespaces + some CLI UX) not too long ago and now you'd be dumb to try and deploy most applications without at least having that there.
How many people are wrangling their own cgroups and namespaces (not counting systemd, which is another thing people fought, tooth and nail)?
Should we all be doing 1 user per workload like the good 'ol days?
Or maybe 1 machine/VM per workload like the 'ol 'ol days?
The latter two are very much variable by company, but those are the general trends. They'll also vary in importance to different people. The former item, however, can be just as burn-out/tedium prone as tech company stack whiplash.
It does help - as you point out - to be in the position where the non-tech company will almost always take your advice if you tell them they need to change platforms. But the most effective way of getting yourself in that position is to prove that you exercise that option only when it truly is in their best interests.
But then again, I like my own yard and sitting by my fire pit with a beer, and I don't spend too much time worrying what other people have.
And so will be the workload. No semi-mandatory after-work drinking, no constant "hustling", no bullshit.
But it's easier to get a company to upgrade from old cruft to a commonly-used technology, of your choice and which you push, over trying to get the company to drop a new fad they've bought into ("You know, we don't really need all of those Docker images, and VMs, and Amazon Cloud services - let's just write an executable which does the thing and can be configured.")
Some possible factors are that the company is not that big and we have a feature list to deliver. We are not VC, so we cannot afford to burn money. The devs churn is low, so we don't need to be attractive to many people all at once.
TBH, I don't like it all the time, but it works and our clients are happy with the results.
Out of the blue, one of your main suppliers declares they will accept orders only through a REST interface starting next quarter. And that you can/must use it to track your orders, instead of sending an email to one of their employees.
One week later, you are required to provide invoices in PDF instead of just printing on paper.
And your accounting system will switch to Oracle Finance in October...
Your ERP was developed on Progress ABL, and nobody really cared to port the old code to a web-based architecture because it was just for "internal use" and Putty was fine.
And maybe these were introduced in other companies because their IT depts. are run by "fashion of the month", but whatever the reason, you now have a problem. And if your sw and your skillset has remained frozen in jade since the early 90s... this will be painful and probably require at least one false start before getting it to work reliably.
I have noted a tendency in people who work always on "very stable" technologies to go in crisis mode as soon they have to (for example) produce a file in XML or JSON format after 2 decades of CSVs... this unfortunately almost routinely ends in stuff which is very brittle ("XSD? you do not understand, we need to produce an XML, ... I dunno what an XSD is or has to do with this requirement - except that they both start with an X, maybe - so please do not make things unnecessarily complicated and just go back to coding, we were supposed to start testing one week ago.")
Implementing orders in REST was probably done in a couple of weeks of work. PDFs have been there since ever.
That is not always true, it also depends on who they are selling code do..
I know a few Orgs that by code from others, and pay $$$$ to ensure it never changes... or rather only changes in measured approved ways not drastic changes that would up in business workflows.
So I would amend your statement to where people write code for companies that sell code to startups or technology companies
if you are selling code to a 100 year old manufacturing company, or some other legacy space (healthcare, education, etc) then most likely they want stability not the new shiny
It's been this way forever. For every level of the stack. When I started the physical hardware hadn't been settled (coax, ethernet, token ring etc.)
Fortunately when I started we only got paid when we shipped. So we were motivated to ship, not be distracted by rewrites. We were also lucky that as a boot-strapped company we had no managers calling the shots. We made decisions and stuck to them, and it's paid off over the long run.
Of course the product has changed immensely over 25 years - but the core code laid down in 1996 is still integral. We've spent the time on the product, not worrying about the sexiness of the tech stack.
Which is not to say we've ignored the tech stack, it's evolved a lot, but the stack serves the product, not the other way around.
On the other hand we don't have to raise investments, we don't have to attract endless streams of new programmers, we don't have to juice a stock price, so we don't have to appear modern or hip. We can just keep doing what we do, making money, focusing on the value to customers, not being (easily) distracted by the next shiny toy.
And yeah, after a couple decades you learn to see through the hype a bit and distinguish things that are good for you, not just the flavor of the week.
Well, some things did bring improvements, or new ways of thinking about existing problems, and some things really moved the state of the art. Taking your example of token ring -> coax (with terminators,eek!) -> ethernet is arguably a good progression (and now also wifi).
But there's also lots of churn in software and people solving the same problems in slightly different ways.
However ... this is also collateral damage from another improvement, imho - that is the ceding of control of the tech landscape from big companies (IBM, Oracle, Microsoft, Intel, Sun, Borland) to smaller open source clusters of people and VC funding. There was a time when nothing happened without some big tech company behind it, and everybody waited for (particularly MS) to bless things and provide library support, IDEs and tools.
Those days are over, but the result is a bazaar of nosql servers, messaging servers, languages and front-end JS frameworks.
There's a whole lot more to modern SWE roles than just staring at an IDE with the job "implement a function that given X returns the square of X" type stuff that current AI can implement well.
I'm not young, yet I don't find any of this all that bad.
Maybe I'm an outlier, but I see refactoring and learning new tools as just keeping the blade sharp. Migrating to new frameworks, tools, etc. is just superficially iterative. I've yet to see some kind of massive paradigm shift blow me away and stress me out.
The problem is not with massive paradigm shifts... actually we are yet to see something like that. The constant busywork on industrial scale is what the problem is, the constant reinvention of the wheel in a way that is somewhat still incompatible yet the same thing. Most change is only for the sake of change, for NIH-syndrome. when a solution for a problem gets half-decent, and almost as good as it was in the windows 2000 times, then it gets forked/dumped and a new half as god fad appears...
There is no problem with learning new tools, but learning a new UI framework every 2 years just to be constantly half-as capable as Delphi was (okay, with rounded corners and animations, I give you that) can be really tiresome.
Cloud vendors changing their line-up constantly so that your certifications expire, and they can remove the really good value for the buck services and push the shiny new globally replicated and 10x more expensive new stuff (which delivers value for many, but the barebones stuff also did for many others).
Just my 0.02$, probably saw too many lets rewrite in $current-thing and don't use $industry-standard-battle-tested-thing, as it is hard to learn it, but let's write one from scratch! (last time a team in a job decided that configuring HAProxy (my suggestion to get stuff done cheaply and quickly) is too hard, it is better to write a high performance reverse proxy, because it will be our thing! (I guess they were more interested in their resumes than in the company's interests). Since then that company is bankrupt... who would've thought)
Replacing weapons and keeping blades sharp is not the same thing. Now, the metaphor kind of breaks down because new weapons are sharp, but abilities with a new technology are typically not, i.e. you use new tech as somewhat of a blunt instrument until you learn the finer points of it - and until it develops and matures enough itself.
So, if you want to use "sharp blades" - don't switch technologies, but rather keep with the updates to those you already know.
Easy to deploy and also you can utilise cheap spot instances to fullest.
I started in web dev almost 20 years ago (did some Visual C++ before that). First job was everything XML. We had our open source CMS that stored all documents as XML and used XSLT to transform it into the HTML we needed. Was pretty cool, but there's always something that didn't fit that paradigm. Some coworkers went a bit overboard and used XSLT to generate the XSLT to create the HTML. But it all works as long as you cache aggressively.
Second was Runy on Rails with JQuery. And eye-opener for me. Very cool to work with. Third job was similar, but using Groovy instead of Ruby. After that, I ended up in SPA-land: first Angular, then Vue, now React. Backends in either Java or Node.js.
I'm still doing that, only now everything needs to be in Docker and Kubernetes.
I still don't know why I should ever need microservices. Our GraphQL. We actually considered GraphQL for previous project, and decided not to because it looked way too complicated. It sounds like it might fit with using a Graph DB (I'm a big fan of neo4j), but it doesn't. At my current project, they made that decision before I joined, and they've since realised that they don't need GraphQL after all, but it's in there now, so they just work around it.
Some of the hypes worked very well for me (XML, jQuery, Ajax, SPA), others seem to work well for others but not me (docker, kubernetes), and some I really don't see the point in (microservices, GraphQL), but I assume there are use cases where they shine.
But don't blindly adopt everything. First check if it actually fills a need you have. And how it will fill that need.
GraphQL from what I can tell supports web developers rewriting their APIs when they need to instead of waiting on back-end teams to support them.
My impression is that GraphQL is great if you're building an api that's going to be used by lots of different teams with different needs and the api developer doesn't know what they are. Again, with a single team doing everything, there's no need for it at all. But we're using it anyway.
Also, they keep talking about a Single SPA Application, and I can't tell if that's just a redundant way to say SPA, or if it's really something new, or if they're just messing with me.
This has been my experience as well. It provides flexibility to the consumers of the API to structure things how they want and fetch only what they need. E.g. if you add a new field, there isn't any impact on existing users to make any changes since they are not specifying it in their current request anyway. To certain extent, that also helps with things being somewhat future proof because even if you knew what the users want today, you can't predict what they will want tomorrow.
Having done a bunch of REST and GraphQL APIs, the only thing I would say for GraphQL is to avoid re-inventing the wheel and use a stable third-party library to do as much heavy lifting as possible so that you can focus on the logic side of things.
Docato/XHive? Been there as well.
Does anyone have any suggestions on anything to help me keep up with the basics of new fads?
I would love a monthly (doesn't have to be exactly monthly, but the point is occasional) magazine, which was the equivalent of those old magazine with basic tutorials, which would give overviews, and comparisions, of all the various modern techs floating around.
I do occasionally find a new person's blog to subscribe to, but I've never found a good general source.
We needed windows machine because of corporate policy and remoted into a Solaris box to dev, as a vague replica of production.
Fast forward to today, and we can run a production replica on our local machines at home, the CPU architectures aren't even the same, and code will be deployed through the CI automatically after the tests all pass.
Sure not all changes are meaningful, but in the grand scheme of things the field is moving forward in spades. I certainly wouldn't want to go back a few decades.
And someone we squandered most of the CPU performance increases in the process, and the super-duper architecture/CI/delivery pipeline fails almost the same percentage of the time, but needs 10x the people to keep it running...
I'm with you on the CPU gains going down the drain and 10x more people working on it, but it's also a sign that we're talking IT operations more seriously and there's enough money to cover these kind of inefficiencies.
A decade ago you could deploy an internal PHP app to a box and forget about it for a few years, then fix a minor bug by sshing in, viming a file, and hacking it a few times.
These days, if you leave the app untouched for six months that 10 minute fix becomes 2 hours of fixing a broken docker build. :)
You also had to keep unholy combo of php extensions (why that garbage is in app server and not app) and server config because the special little retard that designed the app decided he must do something different.
And forget about ever leaving apache, if it doesn't use some apache module to fix PHP/developer ineptitude it will at least have half of the request routing logic in .htaccess file.
No version upgrade required. :D
I tell them that they should let their compliance department know, that the version of PHP isn't even supported, and I list some of the thousand security fixes that they aren't using.
I usually end up VPN/FTP:ing into a forgotten era to add the fixes and I definitely don't want to go back.
No, we didn't update PHP unless strictly needed.
Setting up monitoring for your docker containers is also a whole thing. :)
I think you’re taking my example a little too literally. My point is not that docker/k8s/whatever is bad; just that the ‘new’ adds features at the cost of simplicity.
Which is also the point of monitoring a container, which is fairly reliable nowadays, compared to managing your own health check service to ping your box to check if it's alive.
All in all, I think k8s is way too complicated for what it does for most people, but administrating servers was also a complex task to begin with, and the things sys admins were dealing with looked nightmarish to me. Heck, there was a time companies would have their own SMTP in house...
Those sort of attacks have tapered off though, right? Unless you're engaged in a specific feud, or have caught the ire of a social justice warrior, these days we're mostly looking at data exfiltration as the primary goal. That being said, there are a lot of ongoing feuds and active social justice warriors.
In short: the more moving parts you have, the less time you have between major malfunctions.
(Manufacturing and hardware world understands it well, which is a big part in why they like integrating things so much.)
So e.g. over in the backend-land where I live, it used to be that I had to occasionally update the compiler or one of the few third-party dependencies that I used. Today, I have many more libraries (to the point there's something to update for security reasons roughly once a month, on average), and on top of that, I have CI/CD introducing its own mess, Conan updates which occasionally get messed up, or make some existing recipes incompatible, CMake updates which are done unexpectedly and break stuff, now also Docker is adding more of its own problems, etc. So I have to deal with some kind of tooling breakage every other week now.
And always, always, when I think it's all finally OK and I can get on with my actual job, some forgotten or hidden component craps itself out of the blue. Like that time our git precommit hooks broke for me, because someone changed them in a way that doesn't work with my setup. And then me wasting a day on trying to fix it, eventually giving up and degrading my setup to unblock myself. Or another day where, for no apparent reason, some automation that made automated commits to some git repos started losing Change-ID headers in commit messages, making Gerrit very sad, leading to several people wasting a total of several person-days trying to fix it. Etc.
There's always something breaking, the frequency of such breakages seems to be increasing, and it's a major source of frustration for me in this job. Which is why I too am often thinking back to "good old days", and am increasingly in favor of keeping the amount of dependencies - both libraries and tooling - to a minimum.
I can say with great certainty: Almost no one rebuilds their damn container anywhere near as often as the the gray-beard in the basement updates the Debian packages on the the server that runs the container.
I had the debate with a client, we did monthly security update, unless something horrible happened. The client was rather upset that we didn't patch more frequently, like weekly or daily. My argumentation is that it doesn't really matter if the Linux kernel or bash is patched, when the only thing running is a container with a beta version of Tomcat that hasn't had security updates applied in three years.
Even worse are the people who just pull things from Docker Hub, with no plan as to how and when they'll pull newer versions. But fine, let's just keep running KeyCloak from 2017, and that old Postgresql image which the developer never configured to do backups, I'm sure it's fine.
Most probably the gray-beard simply enabled "unattended-upgrades".
You can do something similar with a container (track the security fixes for the packages used, force rebuild and deploy when needed), but it is a bit of work and I don't know of any ready solution.
How does Kubernetes, microservices and front end frameworks fix that?
They don't. In fact, as someone who works at the more modern end of these things, I'd suspect that "no meaningful monitoring" probably correlates quite well with cloud era microservices thingies.
But we can't blame the tools, that space is just younger and by nature contains more immature stuff.
I don't think the argument here is for keeping things out of version control and monitoring. It's that you can actually leave stuff alone for several years, come back and find it working just as before, when the layers underneath doesn't constantly change. That can be liberating.
I think the article's point was that the industry is perpetually young and immature because it keeps (unnecessarily) chasing fads.
Doesn't being in a container mean its much easier to rebuild it from zero and thus allowing security updates to be more regular?
There's no benefit to using containers here, only more pain.
It’s really difficult to pin everything. Even if you manage that, upstream resources change/move.
Yeah observability is great, but no one I've seen in cloud manages to launch with it. So in place of SRE org with full observability stack, they launch with effectively nothing.
Strong attitudes of "we don't measure hardware metrics like cpu/memory/disk, we measure customer facing stuff like throughput/error rates etc". Sounds good, but what do you think happens to those if you run out of disk bro? Pretty sure the running out of disk happens first. Imagine if you could like.. catch it?
Disk: essentially limitless. If disk of VM runs out, node will crash, new node will start. Service should keep running on other nodes meanwhile. If restarts happen too often, throughput/error rate will suffer or cost will rise.
Modern mentality is "let it crash" [0]. Software has bugs. Design so it can crash and scale depending on need.
[0] https://medium.com/@vamsimokari/erlang-let-it-crash-philosop...
And "let it crash" is hardly "My service is invincible!" I'm sure any serious Erlang deployment has seen what I've seen as well, which is a sufficiently buggy process that crashes often enough that the system never gets into a stable state of handling requests. "Let it crash" is hardly license to write shitty code and let the supervisor pick up the pieces. It's a great backstop, it's a shitty foundation.
I haven't used Erlang, but my impression is that it's probably the same experience there?
In my case, IIRC, it was something to the effect of, a lot of our old clients out in the field connected with a field that crashed the handler. Enough of these were connecting that the supervisor was flipping out and restarting things (even working things) because it thought there was a systemic issue. (I mean, it was right, though I had it configured to be a bit to aggressive in its response.) The fact that I could let things crash didn't rescue the fact my system, even if I fixed that config, would strain its resources constantly doing TLS negotiations and then immediately crashing out what were supposed to be long term connections.
Obviously, the problem was in the test suite, we were able to roll back, address the problem, ultimately this was a blip not a catastrophe. I just cite it as a case where "let it crash" didn't save me, because the thing that was crashing was just too broken. You still have to write good code in Erlang. It may survive not having "great" code but it's not a license to not care.
Many software development processes considered as a whole are anti-fragile... mine certainly is, which is a great deal of why I love automated testing so much (I often phrase it as "it gives me monotonic forward progress" but "it gives me antifragility" is a reasonably close spin on it too)... but that's not unique to Erlang nor does Erlang have anything special to help with it as everything Erlang has for robustness is focused at run time. You can have anti-fragile development processes with any language. (The fact that I successfully left Erlang for another language entirely is also testament to that. I had to replace Erlang's robustness but I didn't have to replace its anti-fragility, since it didn't have it particularly.)
It's not the tool, it's how you use it.
However there are industries applying these lessons in HPC / data analytics / things that touch money live .. operating on scales of users in the 10s to maybe 100s. So stuff where downtime is far more costly both in dollars and reputation.
I'm also intrigued by the constant cloud refrain of "stuff crashes all the time so just expect it to" coming from a background where I have apps that run without crash for 6 months at a time, or essentially until the next release.
I'm all for scaling, recovery, etc.. I just fail to understand why it is desirable for this to be an OR rather than an AND.
What if stuff was highly recoverable and scalable but also.. we just didn't run out of disk needlessly?
IMHO, those aren't mutually exclusive. Your app code should be robust enough to run 6+ months at at time, and the "stuff crashes all the time so just expect it to" attitude should be reserved for stuff outside your control, like hardware failures.
It pertains to e.g. single database connection, single HTTP request etc. If something crashes there your APM reports it and Erlang's unique runtime continues unfazed. It's a property of the BEAM VM that no other runtime possesses.
"Let it crash" is in fact "break your app's runtime logic to many small pieces each of which is independent and an error in any single one does not impact the others".
Scaling an Erlang node is very rarely the solution unless you literally run out of system resources.
RE: error handling, Elixir apps I coded and maintained never had a problem. Everything was 100% transparent which is another huge plus.
In general CGI and PHP had the right idea from the start but the actual implementations were (maybe still are? no idea) subpar.
Erlang's runtime is of course nowhere near what you will get with Rust and very careful programming with the tokio async runtime, but it's the best out there in the land of the dynamic languages. I've tried 10+ languages which is of course not technically exhaustive but I was finally satisfied with making super parallel _and_ resilient workloads when I tried Elixir.
For a lot of tasks I can just code in the Elixir REPL and crank out a solution in an hour, including copy-pasting from the REPL and into an actual program with tests. Same task in Rust took me a week, just saying (though I am still not a senior in Rust; that's absolutely a factor as well) and about 3/4 of a day in Golang.
The only other small-friction no-BS language and ecosystem that came kinda close for me is Golang. I like that one a lot but you have to be very careful when you do true parallelism or you have to reach for a good number of libraries to avoid several sadly very common footguns.
Please read the last three paragraphs in [0]: "a well-designed application is a hierarchy of supervisor and worker processes" and "It handles what makes sense, and allows itself to terminate with an error ("crash") in other cases."
I've personally designed and co-implemented mission critical real-time logistics systems which dealt with tens of thousands of events per second, with hundreds of different microservices deployed on a cluster of 14 heavy nodes. Highly complex logic. At first we were baby-sitting the system, until it became resilient by itself. Stuff crashed all the time. Functional logic was still intact. Then we had true silence for months on our pager alerts.
I call it anti-fragile and Taleb is right. You can't make a system resilient if you don't allow it to fail.
However theres lots of cloud people who don't know linux.
Therein lies the challenge. Too many of these cloud devs forget that whether its "server less" or not, theres a .. server, somewhere.
All the layers of abstraction work when they work, and when they don't .. best of luck figuring out what's going on in a timely fashion. Hope you aren't running any money on top of it.
* microservices help security because for each service you write, you create a minimal surface area (as opposed to monolithic applications where each method has access to maximum dependencies).
* microservices also help monitoring, because failures can be contained and one application does (should) not impact another. This means that your focus of monitoring is on application performance, and less on the underlying system architecture.
* monitoring can be a cross-cutting concern. For example, you can monitor all HTTP requests by instrumenting your ingress layer. Similarly, you can support a unified authorisation scheme by bringing this to the ingress layer. Even more, using microservices, you can independently promote services to production (sort of like feature toggles) or do green/blue promotion.
* security of your microservices is improved, because you can independently upgrade your dependencies per service. Many older monoliths are 'stuck' in a certain dependency configuration and cannot be upgraded without a 'big bang' of expensive dependency resolution and testing.
I could give many more examples, but I hope this captures the spirit a little.
I just did a "npx create-react-app text" and I got
"74 vulnerabilities (69 moderate, 5 high)"
40 major security flaws in 10 years sounds almost like a bargain.
Out of curiosity, why is it discouraged?
I’m not really in the React world, but have been fixing up a Create React App for a client recently, so learned a bit more about it.
Yeah, screw version control! Who needs any record of the change you made? If you have to move the code or redeploy it somewhere because that server goes down or something else changes, _real_ 10x rockstar ninjas remember every change they've made for the last several years.
All other considerations are secondary. Including whether or not your peers approve of your code.
Always remember: the User is king. Programmers are nothing without users, no matter what technology they apply to the situation.
Sounds like you've shipped, and shipped often. GOOD!
Director of O(o)ps: Funny that you should mention that...
> I don't even know how I would do it.
Depends on the software you're writing of course, but the core would probably be some libraries with C (and maybe other language) bindings, some command-line executables, and anything else - above that.
More often than not, they aren't able to properly articulate the why and the trade-offs involved. Sometimes they do fully understand the trade-offs and that leads to good discussions.
I think the computer industry is in its infancy and we haven't really found the right solution to many problems, for everyone to adopt the same thing requires it to be a very good option. BSD sockets is used by everyone - it's that good.
It reminds me of xkcd #927 about standards ( https://xkcd.com/927/ )
I would like a solution to the problem where library code is changed and it breaks code that relies upon it.
I would also like an API description format that we can code against and doesn't break.
Trying new technologies means you're mostly becoming better at using someone else's APIs - this is the path to eventual burn out as the churn continues.
There is a better path - See through the hype. Ask those around you what's the downsides to this approach? And you'll often get blank stares... Why? Because they don't know either - And if they don't know the downsides, they don't really know. They are following the hype curve.
Focus on the fundamental engineering principles and asking better questions - take the bottom-up approach and the reward is you'll find teams that aren't taken by the hype curve so easily.
PS. There are a lot of good technologies that come out, but staying behind the hype curve a little helps you make better judgements over time.
As others say, learning the fundamentals will, and trying to get some more general skill out of every tech fad.
(although, full rewrite every 6 months is excessive imo)
In last 5 years, I have worked at 2 Faang and a large hedge fund and everywhere they had extremely outdated stack or undocumented in-house stack or outdated undocumented in-house stack. Already derecated dependencies were pinned years ago and management won't let you modernize anything since there "no business value in it" and we need to ship a feature instead and if you still insist to refactor and modernize and something goes wrong, now you are personally responsible for the outage. The effort you spent successfully upgrading decades old stack won't count in your promo packet.
There is just no incentive to upgrade anything beyond security fixes. Rewrite to chase the latest fad is frustrating but using technologies which don't even have online documentation anymore is equally if not more frustrating.
Of course the dickhead architect would use articles like this OP as justification for why all of this wasn't a problem.
And I've worked with all of these. The newer stuff is better in every way except penetration (which is held back by bias and misguided evaluations of the word "safe").
PHP at least attempted some isolation between requests (emphasis on attempted) even though it didn't do it very well.
All that being said, people don't like seeing their favorite tech called out and the kneejerk down arrow presses are expected. Programmers are a tribal bunch.
Personally I've found my productivity multiplied with the above-mentioned more modern tech: Elixir, Rust and Golang. But I am also aware many companies would never risk it, and that there is no shortage of people who are OK coasting on old stacks if it pays the bills. Not judging.
Maintaining some existing project is going to get you nothing if you do it right and will bring you problems if it breaks despite your best efforts. Rewriting an existing system in a new tech stack counts as exciting innovative work for promotion purposes, or at least prepares your CV for your next job hop. With salary bumps between levels being what they are in software, I'm not surprised people choose the more lucrative option.
Hell, I have seen the creators of the problems get endlessly promoted for constantly solving problems they created.
Small or medium company, critical systems going down a lot...give me that shit. Easy money and reputation.
Smart businesses and developers instead step-by-step refactor and upgrade their existing apps.
Long enough for the rough edges to be softened, to show that people are willing to maintain it, for detractors to have pointed out legitimate issues, for the tech to become stable.
I’d prefer 10 years personally, but I value stability more than most. I dislike revisiting problems I’ve already solved.
The better understood, both in capability and limitation.
The better chance no new updates will break it.
The better chance no new updates.
The better you get to know it.
The better you get to integrate it.
The better the chance it will be there tomorrow too.
The better the investment learning it becomes.
And also you don't get to integrate it very well if everything around it isn't maintained, and compatible on old version of a library (with said documentation not existing anymore).
Also, while resume-driven development is not a good thing either, it's certainly not a good investment to pigeonhole your career into working with something you hate working with.
But remember to use the right tool for the right job - older isn't always better. E.g. there is a set of people who use stored procedures for everything and it does make sense when you are doing data-heavy processing on database-side. But using that for everything, even when the volume is absurdly low, is a bad idea - using a language like Java would be a million time more flexible, composable, testable (unit tests, anyone?) and allow making changes with greater confidence.
CICD/devops worked because it drastically reduced the risk & expense of defects, the cost of development, time to market, etc.
But the chasing of new technology fads vs lagging behind with old outdated technologies? I am seeing people make decisions without thinking about the decision.
What economic benefit does the new tech bring? Whats the economic trade off of implementing it vs other projects? How long till a return on investment? Whats the costs if we do nothing?
It isn't even that hard to do, but people just keep deciding to spend time and resources with little to no thought. It infuriates me.
Citation needed.
You'll consistently notice obviously idiotic decisions rooted at decision makers who aren't around to own up to the mistake.
The other end has decision makers never change, they waited too long, were too risk averse to stagnate and the cost of change just keeps rising.
The ones who end up surviving this churn, have both kinds of decision makers in equal powers, mistakes are made, but things keep moving sometimes in leaps, sometimes backward steps.
I work in such an environment and routinely deliver million+ line applications with ZERO bugs in production to very large customers around the world. Highly satisfying.
Take for example Rust, which is gaining traction in parts of systems programming. It addresses the direct need to stop overflowing buffers, create more strong typing, write less code to accomplish the same thing, and others. Accordingly, a lot of systems programmers are optimistic and excited about it. But everyone in the field knows C, and acknowledges C isn’t going anywhere. There are tasks for which C is better suited right now, and will be for a very long time. There’s no push to rewrite all programs in Rust. Sure, some are pushing for it, but it’s not a “this year is Rust, next year is <new programming language>” dynamic. It’s a “this is where we want to go, but it’s going to take us many years to get there” type shift.
Things will be tougher when management starts asking for real, measurable results.
And you're supposed to already know how to do it and give an accurate ~~estimate~~ prediction for how long it's going to take to do (that matches the amount of time they've already decided you have to do it). You're a _professional_ and professionals don't need time to actually learn how to do something new, they just know.
All I can give is some advice from my lengthy experience.
Most of the changes happen on the "edges". Those edges are primarily three fold:
First, the user interface: everything that deals with input parsing and formatting. This also includes protocol marshalling (Http, JSON, XML, ...)
Second the data persistence. This involves choice of database or any other persistence. Including serialisation of your data. Any I/O really.
Third, anything to do with deployment. This includes environment isolation, configuration management, process automation and so on.
These the edges have nothing to do with your core application logic.
You can reduce the impact of a switch to a new stack by inflating your core logic and shielding it from the rest of your stack.
This is tricky and I still catch myself all the time doing it wrong. Because doing it wrong is tempting and easy.
For example, consider a connection to the db: you will need a connection string. You can fetch that from an environment variable somewhere deep in your core code. But by reaching out to the system like that, you make that core piece of code dependent on the deployment strategy. Instead, you can move that part as close as possible to the entry point and then pass it down as function/class arguments. You can group similar operations together, close to the entry point. That way, if the dependent e strategy or the stack changes, your core code does not need to change.
Another example is the case where your core code is dependent on a library from your stack. For example, when working with python and flask, it's easy to import "request" or "g" from the framework. But every piece of code relying on that will be tightly coupled to the framework and subject to change whenever the framework changes (either same framework with breaking changes, or framework switch). The solution is the same. Move those elements as close as possible to the entry points. In this example of flask, keep this as close as possible to your routes. But also keep your routes clean of core logic. Interact with the HTTP layer only and pass the values as arguments to your core.
Finally, a similar situation, but much more sneaky and hard to spot is reliance on data types from your stack. Spotting cases where you rely too much on library modules can be done by investigating imports. But reliance on data types is harder to spot. Sticking with the flask example: the http headers are encapsulated by an object of the flask stack (werkzeug in this case). You may be careful and extract the headers in the route to follow my earlier example. But if you then pass the headers unmodified down into your core logic you will again be coupled to the stack. Instead you should extract only the values you care about and hand only those down to break the coupling.
Finding the right balance where to cut the coupling is challenging. The more you cut, the better you will be shielded from changes. But the less you can benefit from pre existing implementations from the stack. Where to cut is ultimately a design decision.
When done well, it will be far less frustrating to jump on the bandwagon of the latest tech. Not completely painless but far less stressful. Not least because you will have confidence that you didn't accidentally break core logic.
First, is that I dove farther into CS fundamentals. This has made new fads a lot easier to deal with because usually they're recycling old CS ideas. React hooks is a good example. I wrote class components for many years, adopting to hooks took me all of a day because Algebraic effects were something I understood well.
Second - I am firm about which tech I do and do not want to learn the specifics of. I understand the high level of kubernetes but will not be learning specifics.
Capitalism doesn't work unless you have something to sell, and that's just as true for tech as it is anywhere else.
But when I started working in the industry, I realized that it's absolutely exhausting. Hype after hype, fad after fad, modern after modern, refactor after refactor. I have a workflow, I know how to build apps. Then one day director of Ops comes and completely and utterly changes the workflow. Ok fine, I'm young, will learn this. Month passes, it is now Terraform. Ok fine I'm young, will learn this. Now we're serverless. Ok fine, will learn. Now everything is container. Ok. Now everything microservice. K. Now turns out lambdas aren't good, so everything is ECS. OK will rewrite everything...
Look I'm not even complaining. But it feels like I'm stuck in a Franz Kafka novel. We just keep changing and changing the same things again and again because that's the new way to do. Big distraction. Destroys your workflow. Forget about all the util scripts you wrote last 6 months being useless.
I don't even know how I would do it. Maybe I would do this the same way if I had any power. But that doesn't change the fact that it's a bit ridiculous. Fun but tiring. Entertaining but exhausting. Cute but frustrating.