Would love to learn more about your setup. Email me if you're interested in chatting. Email in my profile.
I began my career by scripting Asterisk using Python. I created a daemon using Twisted and that process ran for 3 years without fail (as in, the same Linux OS process was running for 3 years). This was on commodity hardware.
So, just make sure you take backups pretty regularly because you never know.
This is a great point. Hyper focus on optimizing micro-containers is costly and sucks up a lot of time. You can still deploy like this in the cloud though. On AWS make a three-year reservation of a c6a.32xlarge to get 128 vCPU for $16k annually. Hetzner is cheaper, 80 vCPU for about €2,506 annually.
They get big discounts on everything you'd pay retail, and their margins for barebones VM is not that large.
I think that brings the ROI after 5 years compared with AWS to something like 3200%, less energy, if nothing goes wrong.
(I know that there are better options than running Windows Server on this box, but it seems to perform the same, it's far easier to administer by inexperienced devops, and in the long term it's actually around $1500/yr cheaper than running supported RHEL. Our other box is running supported RHEL but after testing Windows Server and having no issues we'll probably switch it for the cost savings).
We recently moved everything in-house and started hosting our own servers. Having physical access to the machine (and the network HW) pretty much eliminated all of our dev ops (1). We expected the HW to fail often and such, but in 2 years the only maintenance we had to do was add more disks to the RAID array and resize the partition a few times. Even though we have redundant everything, nothing has ever failed and the server has never gone offline (except when it reboots every Sunday @ 3AM in a min or two).
(1): How? Well instead of running small containers/micro boxes/services that we could scale up and down, since we owned the hardware it meant that efficient utilization wasn't a requirement for us anymore. We don't really care how much we utilize our own HW, so we just smooshed everything onto huge VMs in two huge 100 core boxes with affectionate names. We used super basic networking features (VLANs, firewall port forward) to setup the network. To simplify dev ops even further, there is only access to the servers from specific ethernet ports in the building.