ghomem
Some years ago I made a Mixxx demo video with a DYI "integrated controller". It demos Linux boot to Mixxx, touch screen, beatmatching and some modest effects:
https://www.youtube.com/watch?v=DjHvW4OsQ2Y
Mixxx devs: if you are reading this... cheers :-)
Add to this, that you can have outages due to operational complexity, which tends to happen in complex cloud native setups.
How to scare a computer engineer in 2024? Invite him to configure a server.
https://logical.li/blog/emperors-new-clouds/
I wonder, though, why the topic is so contentious. This is supposed to be an engineering discipline, not a fight between religions.
But don't forget about the definition of micro :-) This is an optimization problem rather than a conceptual one.
What I feel that is missing on what he calls the "Microservices Premium" is a clear statement that this premium is paid in "ops" hours. That changes the game because of the "ops" resource scarcity.
In fact, the microservices dilemma is an optimization problem related to the ops/dev ratio that is being wrongly treated as a conceptual problem.
This is the simplest analysis I could come up with:
On AWS, fargate containers way are more expensive than VMs and non fargate containers are kind of pointless as you have to pay for the VMs where they run anyway. Also auto scaling the containers - without making a mess - is not trivial. Thus, I'm curious. Perhaps it's Lambda? That's a different can of worms.
I'm honestly curious.
If you are doing something serious you probably want to chose suppliers in such a way that you can demonstrate you have security and business continuity under control. That means you probably want to use RHEL, Suse or Ubuntu, distributions for which commercial support exists.
(Ubuntu is particularly interesting because you can start with an LTS release for free and activate commercial support if business goes well, without changing your processes.)
You can think about this beforehand or wait until customers require some kind of certification and the auditors ask you for your suppliers list + the business continuity plan, among other things. You will face this if you deliver to a regulated market or if your customers are large enough to self regulate this kind of thing.
LTS not good enough? Well, cloud native does not have LTS comittement and Pipy does not provide security fixes separated from logical changes.
Try to keep your Terraform code stable for two years in AWS, or try to understand the lifecycle of AWS Glue versions from the docs. Or trust that Google will not discontinue their offers :-)
I mean, maintaining software is never easy of effortless but I respect the effort done by LTS Linux providers - they sell stability and security for a fraction of what you pay for cloud native.
Absolutely. Adhoc configurations should be forbidden! It is easy to ensure dev env reproducibility when you run Linux. If you have config management your devs can have VMs that subscribe to the same exact configuration that the staging prod and dev environments have. They can literally have a deplpyment server in their machine, as a VM. Since the configuration is stored on a server and applied continuously, it is hard to screw it.
You can achieve this with Docker as well, if the arrangement is not too complex.
The problem, at least in my experience, comes when you start depending on several cloud native components where local emulations are always different from the real cloud env in tiny details that are going to screw the deploys over and over.
Feel free to reach out on Linkedin if you need some more details. More than happy to share.
I am generous in the context for generosity. Turns out that engineering is not about being generous but rather about choosing the most efficient solution for problems that in the end need to be business driven. This requires evaluating requirements, context and tradeoffs. That takes a cold, rational mind more than generosity.
> K8s really does address real problems around deployment and it's very well thought out
It's great where it makes sense. It's less than great elsewhere.
Not everything is SaaS, not everything needs scaling, not everything needs 99.99% of uptime, not everything needs a CDN, not every company is VC backed operating at high risk / high reward, etc, etc. Context is better than ideology. If you read the article I posted you will see that stated clearly.
You Kubernetes people get triggered very easily. I was already lucky to have found several juniors that worked in this kind of thing with minimal training. The 'custom Naemon plugin' is 30 lines of bash and you can adapt it to any monitoring system.
Of course this is scary and complicated. I might consider switching to 'Kubernetes operators', which sounds simpler :-)
For the scope I operate, which is pretty standard Linux packages (PostgreSQL, MariaDB, Nginx, Docker, OpenVPN, OpenSSH) the changes between 16.04 and 22.04 have been quite OK to deal with.
You can use the entire life cycle but not one is forcing you to. You can update from one LTS to another every 2 years, or 4 years, or 5 years... you decide.
So, you had technical scalability in one system but if the customer base grew quickly every other bottleneck would be revealed.
There is more to business operations than technology, it seems.
If I was to write an idempotent script for each native resource I would finish in some years :-)
You chose whatever monitoring system you like the most.
For offline nodes you use whatever the level of criticity of your node justifies. This is something people struggle to understand: not every business needs 99.99% uptime. That said, I never had a downtime in Hetzner. On Digital ocean I had one short forced reboot in 4 years. YMMV so protect yourself as much as necessary.
Deploying on a different provider than Hetzner is the same as deploying on Hetzner except the part of launching the machine which is trivial to script - the added value is making the machine work and Ubuntu/Debian/RHEL are the same everywhere. You don't have vendor lock in with this.
If K8s works for you, enjoy it. Nobody is telling you to stop :-)
This case is actually pretty simple.
Puppet applies the configuration you declare impotently when you run the Puppet agent: whatever is not configured gets configured, whatever is already configured remains the same.
If there is an error the return code of the Puppet agent is different from that of the situations above.
Knowing this you can choose triggering the Puppet agent runs remotely from a monitoring system, (instead of periodical local runs), collecting the exit code and monitoring the status of that exit code inside the monitoring system.
Therefore, instead of having an agent that runs silently leaving you logs to parse, you have a green light / red light system in regards to the compliance of a machine with its manifesto. If somebody broke the machine leaving it in an unconfigurable state or if someone broke its manifesto during configuration maintenance you will soon get a red light and the corresponding notifications.
This is active configuration management rather than what people usually call provisioning.
Of course you need an SSH connection for this execution and with that you need hardened SSH config, whitelisting, dedicated unpriviledged user for monitoring, exceptional finegrained sudo cases, etc. Not rocket science.
I use Puppet for distribution of users, firewall rules, SSH hardening + whitelisting, nginx config (rev proxy, static server, etc), Let's Encrypt certs management + renewal + distribution, PostgreSQL config, etc.
The profit from this is huge once you have say 20-30 machines instead of 2-3, user lifecycle in the team that needs to be managed, etc. But the time investment is not trivial - for a couple of machines it is not worth it.
This does not happen with Puppet + Linux, because LTS distributions have a long release cycle where compatibility is not broken.
I tried to explain this topic in the article linked above. Not sure how far I succeeded.
Pipy is an informal source of software that has low security levels and was infested with malware many times over the years. It does not provide security updates: it provides updates that might include security-related changes as well as functional changes. Whenever you update a package from there, there is a chain reaction of dependency updates that insert untested code in your product.
Due to this, I prefer to target an LTS platform (Ubuntu LTS, Debian, RHEL...) and adapt to whatever python environment exists there, enjoying the fact that I can blindly update a package due to security (ex: Django) without worrying that it will be a new version which could break my app. *
Furthermore, with Ubuntu I can get a formal contract with Canonical without changing anything on my setup, and with RHEL it comes built-in with the subscription. Last time I checked Canonical's security team was around 30pax (whereas Pipy recently hired their first security engineer). These things provide supply-chain peace of mind to whoever consumes the software, not only to who maintains it.
I really need to write an article about this.
* exceptions apply, context is king
s/host OS independence/a certain level of host OS independence
And getting containers to run depends on the OS - if you don't control the host, leads to major ping-pongs.
Even within Linux (Ubuntu, Debian, RHEL, etc) when you are distributing multiple related containers there are details to care about, not about the container itself but about the base OS configuration. It's not magic.
The connection of this to the replies above it: I am not sure if this kind of junior people would be easy to retain in a large corporate environment. We certainly can do that in niche consulting.
For Python backends I often deploy the code directly with a Puppet resource called VcsRepo which basically places a certain tag of a certain repo on a certain filesystem location. And I also package the systemd scripts for easy start/stop/restart. You can do this with other config management tools, via bash or by hand, depending on how many systems you manage.
What bothers me with your question is Pip :-) But perhaps that is off topic...?
In the past years I was solving a data pipeline mess on a project which also had a devops AWS mess. First thing I was told was "what we need is a data lake".
Decisions are sticky so take context into account.
https://survey.stackoverflow.co/2022/#developer-profile-deve...
Only around 11% of the whole devs identify as devops specialist or cloud infrastructure engineer.
This is why I am saying ops people are a scarce good (unfortunately) from a data driven perspective. Of course my daily life confirms it.