News for hackers. Stuff that matters.
Re:Lessons Learned from 11 Years of Hosting a SaaS by Radim on Thursday June 15, 2023 @02:32AM (#36337110)
It may be a generational thing, a matter of familiarity with computing and computers. For someone who's lived through the 80s, "handrolling postgres" doesn't sound nearly as scary as you imagine.

I expect the cost/benefit analysis of "handrolling postgres on a beefy Hetzner server" vs "navigating the menus and options of AWS services" would be different for different people.

Re:Lessons Learned from 11 Years of Hosting a SaaS by erhaetherth on Thursday June 15, 2023 @02:36AM (#36337141)
I've been on unmanaged MySQL for ~8 years now. Considered switching to managed but I'm not seeing any performance or stability issues, so I guess I'll just keep this train going until it craps out on me, then restore a backup onto a managed service, say sorry for the downtime, and that'll be that.
Re:Lessons Learned from 11 Years of Hosting a SaaS by whynotmaybe on Thursday June 15, 2023 @07:24AM (#36338907)
Do you know how long the downtime might be? Have you tested your backup recently?

Gitlab had a long downtime because the backup was huge and on the other side of the country. The backup server was on a low speed network.

https://www.arcserve.com/blog/lessons-learned-gitlabs-massiv...

How much money would you lose if you were down for one week? How many customer would you lose?

How much credibility would you lose?

For my peace of mind, I can't afford a spof when I know one lingering.

Re:Lessons Learned from 11 Years of Hosting a SaaS by tacker2000 on Thursday June 15, 2023 @11:06AM (#36341559)
I get what your saying here, but its again the comparison with Github and extremely large sites thats the problem. Most of us dont run google/fb/github scale sites and the backup will probably fit on an external HDD and in some cases would be even downloadable over S3 in an hour.
Re:Lessons Learned from 11 Years of Hosting a SaaS by whynotmaybe on Saturday June 17, 2023 @12:56AM (#36367425)
That's what I can't be comfortable with : "would".

How long does it take to try it? A day?

Well then try it, either it'll work flawlessly on the first try, either you'll learn that the backup you have doesn't include logins, password and the security configuration that goes with it. Or that the dump you took lost some data because it wasn't in the right encoding.

Or the tape drive you're using need specific drivers that aren't available on the web anymore because the company website's closed.

... This is a work of fiction. Any similarity to actual events might be purely coincidental...

Re:Lessons Learned from 11 Years of Hosting a SaaS by raverbashing on Thursday June 15, 2023 @02:46AM (#36337200)
I agree

And more to the point, learning how to "handroll" Postgres could be beneficial. You could have learned about options for limiting the amount of memory, etc

Sure, managed is easier and use it when you can afford it easily. But before that, it's better to see how things are going (mem usage, disk usage, bottlenecks, etc)

Re:Lessons Learned from 11 Years of Hosting a SaaS by speedgoose on Thursday June 15, 2023 @03:18AM (#36337403)
I'm looking at the ansible playbooks to setup my favourite beefy baremetal Hetzner server (128GB ram, Ryzen 9 5950X 16-Core, 450Gb fast NVME SSD, 3.5TB x2 NVME SSDs, 155€/month):

- Install Debian 11 while booted in rescue mode.

- Setup the root file system encryption using cryptsetup and dropbear (to enter the key during the boot through SSH). Involves chroot and some fun commands.

- Setup ZFS encrypted mirror filesystem for the two additional SSDs.

- OpenSSH hardening and Teleport installation.

- Kubernetes installation (K3S)

- Connecting kubernetes to my Argo CD instance or an existing Kubernetes cluster.

And then through GitOps:

- Installation of openebs-zfspv

- Installation of kube-prometheus-stack helm chart

- Installation of (many) postgresql instances and other craps

I have been playing with Linux servers for 20 years and I find this fun and rewarding. But I do understand people saying that baremetal Hetzner is not for everyone. Especially if you start to have requirements such as "data must be encrypted at rest".

Re:Lessons Learned from 11 Years of Hosting a SaaS by gog on Thursday June 15, 2023 @03:43AM (#36337530)
If you skip Kubernetes the setup is not that complicated.
Re:Lessons Learned from 11 Years of Hosting a SaaS by speedgoose on Thursday June 15, 2023 @05:10AM (#36338021)
I'm not sure. I rather use a lightweight kubernetes or perhaps nomad than do everything that kubernetes does without it. It sounds even more complicated. But I agree that for one single postgresql isolated from everything, kubernetes is overkill.
Re:Lessons Learned from 11 Years of Hosting a SaaS by didntcheck on Thursday June 15, 2023 @06:34AM (#36338559)
Yes, that list reminds me of the exaggerated posts about "look how hard it is to install Firefox on Linux!!". Claiming that setting up a Debian Postgres server necessarily entails knowing ZFS and Kubernetes is quite a reach

Not sure about Debian, but I believe Ubuntu Server will let you setup an mdadm mirror, LUKS (with LVM), and install and enable a Postgres server with a few buttons in the install wizard. It can even fetch SSH authorized keys from a Github account, covering by far the most important SSH hardening step (disabling passwords). Most hosting providers will also offer a one-click deploy that may similarly add your keys and do other common config

A better example of something that hosted databases makes a lot easier out of the box would be backup, replication, and monitoring

Re:Lessons Learned from 11 Years of Hosting a SaaS by marginalia_nu on Thursday June 15, 2023 @03:58AM (#36337622)
Why are you running cluster orchestration software on a single machine?
Re:Lessons Learned from 11 Years of Hosting a SaaS by speedgoose on Thursday June 15, 2023 @05:11AM (#36338030)
My clusters don't necessarily have one machine, but I have a few clusters with a single node.

K3S is pretty lightweight and kubernetes is much more than cluster orchestration, so the pros win against the cons.

Re:Lessons Learned from 11 Years of Hosting a SaaS by kortilla on Thursday June 15, 2023 @05:40AM (#36338227)
Kubernetes is not a cluster orchestration software. It’s an app orchestration software that also supports multiple nodes. If you don’t understand the difference, you’re missing the value proposition of k8s.
Re:Lessons Learned from 11 Years of Hosting a SaaS by marginalia_nu on Thursday June 15, 2023 @06:43AM (#36338617)
Well, yeah. I'm not sure I do see the value prop of k8s.

The fact that it's a supports multi-node means that you get all of the drawbacks of a multi-node system without any of the benefits. It's single node deployment but worse.

Re:Lessons Learned from 11 Years of Hosting a SaaS by maccard on Thursday June 15, 2023 @04:05AM (#36337667)
Our terraform config for RDS is about 50 lines of configuration. We get a much smaller instance for our money, but ultimately figuring out all of what you posted isn't a good use of my time (yet).
Re:Lessons Learned from 11 Years of Hosting a SaaS by speedgoose on Thursday June 15, 2023 @05:12AM (#36338037)
I agree, if you don't have the time and don't find it fun, managed services in the cloud is a better idea.
Re:Lessons Learned from 11 Years of Hosting a SaaS by riku_iki on Friday June 16, 2023 @01:24AM (#36351747)
and you absolutely can't be sure your saas vendor is doing encryption and hardening.
Re:Lessons Learned from 11 Years of Hosting a SaaS by riku_iki on Friday June 16, 2023 @01:36AM (#36351845)
> - Installation of (many) postgresql instances and other craps

and you missing part about fault tolerance and fall back which is most complicated.

Re:Lessons Learned from 11 Years of Hosting a SaaS by porsager on Tuesday June 20, 2023 @03:04PM (#36408921)
pg_auto_failover has your back - https://github.com/hapostgres/pg_auto_failover
Re:Lessons Learned from 11 Years of Hosting a SaaS by Ambix on Sunday June 18, 2023 @06:45AM (#36379014)
OMG you do not need this bloatload for just PG hosting. Just harden SSH, harden PG configs and voila :)
Re:Lessons Learned from 11 Years of Hosting a SaaS by SanderNL on Thursday June 15, 2023 @03:31AM (#36337460)
I don't think "modern" stacks are sane enough to "handroll" anymore. Sure, you can do it, but look at the poster in this thread that details the setup of a debian server.

Kubernetes, "Argo CD", zero-trust, the sheer amount of "management" is off the chart.

"Installation of kube-prometheus-stack helm chart".. "Installation of openebs-zfspv"..

It's not postgres that's the problem here.

Re:Lessons Learned from 11 Years of Hosting a SaaS by marginalia_nu on Thursday June 15, 2023 @03:57AM (#36337616)
A lot of "modern" stacks is just complexity for the sake of complexity. Google is doing it so clearly our 5 man startup will face the same scaling problems, or something like that.

Many of the problems these tools solve are problems that wouldn't exist building things the old fashioned way. If you stick relatively close to the metal, operating this stuff is pretty easy.

However it's notable that a very valid reason to prefer managed services as a SaaS is to cover your ass if things go wrong. Your SLA violation is their SLA violation.

Re:Lessons Learned from 11 Years of Hosting a SaaS by maccard on Thursday June 15, 2023 @04:12AM (#36337702)
The complex stacks are insane to operate yourself but very simple to operate if you use a managed offering instead, and they do provide genuine value.

I can set up a new golang app on ECS with a load balancer and database, with a CI/CD pipeline, with 0 downtime updates in about 30 minutes. Most of that time is waiting for AWS to give me a load balancer. Our work applications have been running with this setup for over 2 years and the only thing Ivs done with infra in that time is adjusted instance sizes and bumped a MySql version.

Re:Lessons Learned from 11 Years of Hosting a SaaS by marginalia_nu on Thursday June 15, 2023 @04:39AM (#36337840)
To be fair, I can set up a new service on bare metal in minutes too, mostly because I don't need to set up everything from scratch.

I don't really need to set up a database or load balancer or anything like that because it already exists on the server. Just create a new database schema, new systemd service, new nginx rule.

Re:Lessons Learned from 11 Years of Hosting a SaaS by tacker2000 on Thursday June 15, 2023 @09:12AM (#36339958)
Yea i also dont see the point of having everything “managed”.

RDS is crazy expensive compared to self hosting and if i have the DB on prem its much faster as well. And the admin overhead is not so big to be honest if you are using just one DB.

If you are Google scale of course things will change, but I think 80% of loads dont need any managed AWS stuff, replications, multiple nodes, kubernetes, etc… just periodic backups and it runs fine.

But people nowadays just like throwing money around I guess, instead of trying to set it up for themselves.

Re:Lessons Learned from 11 Years of Hosting a SaaS by Ambix on Sunday June 18, 2023 @06:41AM (#36378994)
Yep, I'd always prefer freedom and power of hosting my own PG instance upon a some robust VM offering to guiggling with clumsy AWS menus.
Re:Lessons Learned from 11 Years of Hosting a SaaS by PaulRobinson on Sunday June 18, 2023 @08:15AM (#36379524)
If you're managing AWS infra through the web application, you're definitely not doing things as per AWS-prescribed best practice.