In less than a six-week cycle, we built those operational foundations, shaped mrsk to its functional form and had Tadalist running in production on our own hardware.
You do need lots of expertise and relatively simple applications to replace something like k8s, but 37signals seems up to the task, and judging by the article, they picked their least critical apps to start with. It sounds like a success story so far. Kudos to them for releasing MRSK, it definitely looks interesting.
As a side note, I've become disgruntled at k8s becoming the defacto standard for deploying services at scale. We need different approaches to container orchestration, that do things differently (perhaps even rethinking containers!), and focus on simplicity and usability instead of just hyper scalability, which many projects don't need.
I was a fan of Docker Swarm for a long time, and still use it at home, but I wouldn't dare recommend it professionally anymore. Especially with the current way Docker Inc. is managed.
please provide more insight, thanks
Even mirantis is back supporting swarmkit and adding new features.
I don't get why people in IT need a boogeyman. Looks like "k8s is hard" is the new "pointers are hard" we use to scare other people now.
The issue is not about k8s being hard. Yes, it has a steep learning curve, but many technologies do. The issue is that learning all of its intricacies, and maintaining it and the services that run on top of it, requires valuable resources many companies don't have, especially early on. And the benefits it provides are, for the most part, not needed early on in a project's lifecycle, and often never. In financial terms, it's very high risk, with low ROI.
If there's a solution that lowers the investment and maintenance costs, while being valuable in the short and long term, then that's generally a more favorable solution for most projects that don't operate at Google's scale.
There, what YOU know from experience and have evolved and worked through is unknown—because it is all new. And “training” (such as it is) is left as an exercise for each individual.
I’d expect that is the norm for the traditional non startup firms, globally.
If you are a user of K8s, then yeah, deploying apps is pretty simple most of the time.
Maintaining a K8s cluster on the other hand becomes very complex and hard the moment you have use cases that are a few steps off the happy path. The K8s documentation is not sufficient for operating a K8s cluster on your own hardware, you end up having to go spelunking in the code to see how things work this is from experience).
K8s is not hard, but complex.
When you transition an IT team or a customer facing product support team to DevOps, most everything appears complex if the implementation has been done by engineers new to DevOps and cloud itself. Engineers with zero background in scale out or performance for larger customers. It is a cultural/experience change that faces issues at actual deployment time.
I’m watching that play out at work now.
This is such a key phrase here.
If I'm starting a small SaaS company tomorrow, my ideal for setting up infrastructure would be a stack which can for now look similar to what this article sets up (especially with the tremendously lower bills), but with an easy migration path to k8s, should I hit the jackpot and have that 'very good problem to have' of too many customer requests to handle.
My big issue with k8s, and honestly with other big fancy toolsets, is that getting started with it requires you to choose between:
- Hire several seasoned cloud orchestration experts, preferably with the platform you've chosen (AWS, GCP, Azure) who will know how to troubleshoot this beast when you have a mysterious issue, or: - YOLO it! Just follow the basic tutorials to set k8s up, and hope you don't end up sitting up all night with a site that's refusing connections while your customers flee.
The first one is the only responsible choice but it's going to add another half million to your cash burn, and that's on top of the high-margin "managed" service cloud bills like RDS.
So I can see why people are drawn to a system where instead of paying for k8s and "Postgres in a box" they can pay for a simple server and have simple tooling to deploy, back up, etc.
I was reviewing GCP's Free Tier today, they have the same approach, if they need to change or drop services they agree to give 30 days notice, same as Docker did. It's probably common for other cloud companies offering free stuff as well. All the negative attention Docker received was fully and wholly undeserved.
Google is notoriously bad about this and gets negative attention from it, so the comparison isn't favorable, and the publicity is still wholly deserved.
Do you mean Google or GCP? We don’t see complaints about AWS because Amazon closes Dash buttons or Spark, and also Azure is not seen in any worse light due to Microsoft discontinues Skype and what not.
Can we name one remotely popular service of GCP that has been shut down at all?
Until then, I maintain the Docker publicity is undeserved and if I had to guess, was brought on by podman astroturfers who have been polluting the web the past 2 years claiming how great podman is.
Honestly, its inbuilt horizontal scaling systems are pretty lacking. Scaling is not actually K8s's strong suit - sure, you can make it scale, but that takes effort and customization.
But what K8s, at base, is actually useful for is availability.
You tell K8s how many instances of a thing to run; it runs them; if any of them stop running, it detects that and tries to fix it.
When you want to deploy a new version, it replaces the old instances with new ones, while ensuring traffic still gets served.
And it does all of this over a substrate of shared underlying server nodes, in such a way that if any of those servers goes down, it will redistribute workloads to compensate.
All of that is useful even if you don't care about scale.
AWS ECS -> Google GKE -> AWS EKS -> Self hosted Kubernetes -> bespoke solution.
They are pretty much changing their tech stack every few years, never maturing their knowledge of the current tech stack.
I am sure it makes sense to self host, but given their track record I am wondering if they wil be migrating to the cloud again in a few years.
Maybe 'rolled their own' or 'first invented the universe' would have been slightly better.
But I do feel like they probably know what they're doing enough to have a more modest version of success with this other project, i.e., meeting their own needs well without burning up too much money or time. They're still a really small, focused company, and they have a lot of relevant experience.
And for every Rails out there, there are thousands of internal frameworks with big ambitions that just turned out to be inferior to what's already available. You just can't know it when you start developing. It takes a bit of ego and ambition to go that path, but sometimes it pays off. And my guess is that if it paid off in the past, you're more likely to try it again.
This is not a company that merely shaves its Yaks. It offers a full menu of Yak barber services, and then launches a line of successful Yak grooming products.
After typing that I found that they renamed from Basecamp Inc. back to 37signals and their website is trying to hearken to their past. https://en.wikipedia.org/wiki/37signals
Edit: lol https://37signals.com/22/
later
I added "six years later", but I don't think it changes the meaning of what I wrote originally.
no they didn't
Thanks for subjecting me to this crap article ( Which i presume you didn't bother to read.).
Do you believe that a significant proportion of the staff did not quit? Do you have an alternative source that provides evidence for that version of events?
> Do you believe that a significant proportion of the staff did not quit?
Yeah i'll go around trying to prove the negative when i was not the one that claimed some BS as "fact" ( i know you are different from op).
That seems like a slightly different prior, in terms of our Bayesian assessment of the probability that those people remained employed at the company afterwards, than your hypothetical engagement to Ms Johannsen.
Where did you get this though?
> had announced their intention to accept buyouts from the company.
Is it just people clicking 'yes' reaction to internal slack message ? This didn't sound like they were making any commitment ' to the company' .
Also do you have any comment about the title of the article that you linked. Does that seem honest to you?
> As a result of the recent changes at Basecamp, today is my last day at the company. I joined over 15 years ago as a junior programmer and I’ve been involved with nearly every product launch there since 2006.
https://web.archive.org/web/20210430155528/https://twitter.c...
https://web.archive.org/web/20210430140035/https://twitter.c...
https://twitter.com/zachwaugh/status/1388190748189802501
> I’m leaving my position at Basecamp, where I’ve worked for 4 years, due to the recent changes and new policies.
https://twitter.com/lexicola/status/1388189598367559688
https://twitter.com/dylanginsburg/status/1388199059983413257
https://twitter.com/jonasdowney/status/1388205182916440070
> Given the recent changes at Basecamp, I’ve decided to leave my job as Head of Design.
https://twitter.com/mackesque/status/1388206605506842627
https://twitter.com/kaspth/status/1380616358266871810
https://twitter.com/wcmoline/status/1388208323908968449
> I have left Basecamp due to the recent changes & policies.
https://twitter.com/conormuirhead/status/1388207801646780416
it was just skepticism from seeing these sorts of claims over the years. Half of hollywood would be in canada if people really followed up on those.
I guess quitting your job in the hottest tech market of all time is a little different than moving to a different country.
You are deeply misunderstanding Kubernetes if you think it's some sort of a turnkey solution that solves all your infrastructure problems. Virtually everything of value in Kubernetes isn't Kubernetes -- you have to add it on later, and manage it yourself. Container runtime? -- that's not Kubernetes. Database to store deployment info? -- that's not Kubernetes. Network setup and management? -- that's not Kubernetes. Storage setup and management? -- still not Kubernetes.
When you start using Kubernetes for real, you will end up replacing almost every component it brings by default with something else. CoreDNS? -- sucks for big systems. Volumes? You aren't going to be using volumes from local filesystem... that's insane! You'll probably set up Ceph or something like that and add some operators to help you use it. Permission management? -- Well, you are out of luck in a major way here... you have, basically, Kyverno, but it really, really sucks (and it's still not Kubernetes!).
Real-life Kubernetes deployments end up being very poorly stitched together piles of different components. So much so that you start wishing you'd never touched that thing because a huge fraction of the stuff you now need to manage is integration with Kubernetes on top of the functionality provided by these components.
When you install Kubernetes, you get a container runtime. That's a distribution I guess. Part of this seems like GNU/Linux.
The other stuff you're listing isn't solved by MRSK either...
Using local volumes is actually not an insane idea if your stateful service can handle data replication themselves: many modern databases can.
Obviously, you cannot move local volumes around.
And if you are setting up a database in Kubernetes... oh, you are in such a pit of troubles, that dealing with local volumes isn't really even worth mentioning. Surprisingly, your problems don't even start with storage, they start with memory. Databases really like memory, but use it very opportunistically, and scale well with load. So, when you configure your database, you tend to give them all the memory you have, but when they use it, it will really depend on the load and the kind of queries, how well they optimize it. Since Kubernetes scheduler doesn't really do well with reservations, you may run into situations where your database OOMs or just slows everything down, or doesn't perform well at all...
Next comes fsync. Unlike many unsophisticated applications, databases don't like losing data. That's why they want to use fsync in some capacity. But this creates problems sharing resources, again, well beyond anything Kubernetes can help with.
Next comes provisioning of high-quality storage for databases... and storage likes to come in the form of a device, not filesystem, but Kubernetes doesn't know how to deal with devices, so, it needs a help from CSIs of all sorts to do that, and depending on technology you choose, you'll have a very immersive journey into the world of hacks and muti-thousand page protocol descriptions telling you how to connect your storage and Kubernetes.
It might appear though, at the first glance, that things work well w/o much intervention, and there's a Helm chart for this or the other provider, and it's all at the tips of your fingers... but, as it often is in the world of storage, things get extremely complicated extremely quickly in case of errors. In such situations, Kubernetes will only obscure the problem. Oh, and errors in storage don't usually happen in the next hour or day or even year after you've set it up. It hits you few years later, once you've accumulated a ton of useful data and you've entirely forgotten how things have been set up, and folks in Kubernetes had moved on and broke stuff.
---
So, not only do you need small scale, you also need a very short temporal scale: don't expect your Kubernetes cluster to work well after about a year of being deployed. Probably not at all after five years.
But then... if it only works at small scale and for short time? -- is it really worth the trouble? I mean, Kubernetes isn't a small thing, it takes away a big constant share of your resources, which it promises to amortize with scale. You are essentially preaching the same idea as Electron-based desktop applications or Docker containers that create a lot of duplication of entire Linux user-space + a bunch of common libraries if you aren't extremely careful with that. Doesn't it become an argument for producing hot garbage as fast as possible so that someone else who can do a better job won't get a chance of selling their goods because they didn't have time to deliver?
>> Local volumes don't have a concept of quota. You cannot limit them to X bytes. So, if you give a single service a volume, it might just take the whole disk.
That's why we monitor our server disk for usage.
>> Obviously, you cannot move local volumes around.
Most of the time, this is not a requirements for database.
>> Since Kubernetes scheduler doesn't really do well with reservations, you may run into situations where your database OOMs or just slows everything down, or doesn't perform well at all
Unless it's test cluster with constraint resources, no other services will run on database nodes, through the use of taint and toleration. We can let the database use all the CPU and memory it wants
>> fsync Doesn't matter with local volume, since it's just a directory on the host system.
>> Next comes provisioning of high-quality storage for databases... and storage likes to come in the form of a device, not filesystem We didn't deploy our databases with raw block devices, even before K8S. Using regular filesystems make everything much simpler and we did not see any performance difference.
>> You are essentially preaching the same idea as Electron-based desktop applications or Docker containers that create a lot of duplication of entire Linux user-space + a bunch of common libraries
Yeah, no. If that's how you read it, be my guest, but don't put words into my mouth.
I for one am patiently waiting for more innovation in this area and seeing that there are companies that try to disrupt/improve it makes me hopeful and I appreciate it.
> It all sounded like a win-win situation, but [on-prem kubernetes] turned out to be a very expensive and operationally complicated idea, so we had to get back to the drawing board pretty soon.
It was very expensive and operationally complicated to self-host k8s, so they decided to build their own orchestration tooling? Sort of undercuts their main argument that this bit isn't even remotely fleshed out.