There primary app was Rails, you know what replaced all of it? Running the synchronous tasks on the app servers themselves, the async ones on Sidekiq and changing the max instances on the two ASGs.
The only downside has been not being able to meme about how we're web scale anymore.
However, great job. Once they get to that scale of objects it becomes really hard to clean it up - especially if most the people that wrote them have already left the company for example.
Curious if you ended up also saving money in the process?
Can you be more specific about how serverless tends to turn into a ball of spaghetti? It is simply because the lack of state becomes more of a bug than a feature?
Also for event processing consider the case you need to process events in order for instance. That is a ton easier with stateful consumers, batching etc which while possible in serverless is usually more difficult/off the beaten path.
However by spaghetti I'm more referring to the architecture of your application being turned into networked components, i.e a distributed system.
It's pretty much accepted fact now that distributed systems are by far the most challenging area of both computer science and software engineering.
Serverless doesn't technically -mandate- this, you could put all your code in a giant function. However it strongly incentivises you not to. a) generally it's deployment times and cold start issues get worse with function size and b) it pushes you to adopt other "building blocks" which are themselves network attached.
Essentially it leads people into a massively distributed micro-service architecture right out of the gate. For most applications this is definitely the wrong choice even if you have the folks on staff that specialise in distributed systems, usually those folk are the last to advocate for anything distributed ironically.
Getting into the weeds as for why this is hard and all the problems you can run into is probably a bit much for a single HN comment but if you really are curious I suggest reading Leslie Lamport and John Ousterhout. Both have covered the problem space in depth with foundational papers, Leslies "Time, Clocks, and the Ordering of Events in a Distributed System" in particular is foundational.
> For example it can be way simpler to use a in-memory cache than a network attached cache and by doing so you save yourself a dependency, a potential network request that can fail, etc.
Elsewhere you decided against using cloudflare's kv because it was a poor fit for "cache invalidation" -- "way simpler"? -- and then of course turns out you really don't know if it's a bad fit.
> by spaghetti I'm more referring to the architecture of your application being turned into networked components, i.e a distributed system.
That is a novel definition of "spagetti". It's a pejorative term in software sense. Why?
> It's pretty much accepted fact now that distributed systems are by far the most challenging area of both computer science and software engineering.
Ahem. So it caching and cache invalidation. But that didn't stop you, so exactly how is this line you seem to draw with such vocal authority regarding "bad hard", "good hard" informed technically?
I don't necessarily agree with everything OP is saying, and I certainly don't speak for them, but I completely understand what they mean by a big ball of microservice spaghetti. The spaghetti is distributed across a massive literal network of dependencies instead of being distributed off a giant graph of object dependencies, or goto statements or something. The spaghetti is real and just as messy in the microservice world as the monolith world. I should know, we inherited a massive one.
But this shouldn't be too much of a revelation since there is no silver bullet in taming software architecture. Architecting software is very subjective, and more often than not we make the wrong decisions and have to untangle the mistakes further down the road.
Sure, I understand that too, but that's not what was said. Here it is again:
> by spaghetti I'm more referring to the architecture of your application being turned into networked components, i.e a distributed system.
I should know, I have a ball of networked spaghetti.
I evaluated it at the time. I don't have the context loaded to remember exactly why it didn't work for my use-case but I'm going to look again because CFs platform moves pretty fast and perhaps it could be simpler than what we are currently doing.
> That is a novel definition of "spagetti". It's a pejorative term in software sense. Why?
Elsewhere it's been coined Lambda Pinball, perhaps you find that a more pleasing description?
> Ahem. So it caching and cache invalidation. But that didn't stop you, so exactly how is this line you seem to draw with such vocal authority regarding "bad hard", "good hard" informed technically?
Yes, both of which are generally necessary in distributed systems. i.e not only is it a hard problem in and of itself it requires solving sub-problems which are also some of the hardest problems.
I think you may have made my points for me.
By off the beaten path you mean a few lines of yaml to configure a FIFO SQS queue and have it trigger a Lambda?
https://aws.amazon.com/blogs/aws/aws-lambda-adds-amazon-simp...
1) event-driven
2) procedural
3) functional
4) inheritance
I would say event-driven is tied for last as the hardest to reason about.
Yeah, totally easy!
Different people have different strengths and anxieties. The suggestion that, with some regularity, the lack of (local) ambiguity of event message arrival may be of higher leverage relative to complexity than increasing the surface of an invokable interface does not seem beyond the pale.
Ultimately relinquishing a total temporal ordering is going to be painful regardless of whether it's local or distributed.
What I've seen though is that in many cases it is unsolved for and it's very hard to reason about "if I remove this event because it's 5 versions too old already and I don't want to keep this backwards compatibility layer, will something "at the other end of our system I never even interact with" puke?
Granted some orgs are so big that it makes sense to solve for these large landscapes of services with backwards compatibility ad absurdum in order to allow anything to get done (been at that bank, done that). But reason about the entire system it does not make easier.
At many smaller orgs these services were also not built because of organizational issues that needed to be solved but because "everyone does this now" and there's no proper registry of properly defined interfaces and trying to find out if a change you were going to make is going to take you forever because you do need to version something or if you can just change things as needed is way harder than it's supposed to be at that size org.
I repeat: most orgs that go all in on this are actually so small that being so compatible is more of a drag than understanding the part of the system that is attached to the interface you are trying to change ;)
I could understand making this claim w.r.t. Idempotence. Claiming statelessnes as a requirement strikes me as non-trivial enough to require some evidence or a citation thereto.
Non-sequitur
I didn't end up using KV or other features yet, they weren't a good fit for the cache revalidation implementation.
I think at-least for now I see it differently to FaaS services like Lambda/CloudFunctions because I don't intend to use it to implement application logic, rather just augment (or in this case shore up deficiencies) in the edge.
We may consider doing more stuff at the edge later for latency advantages. This to me changes the calculus because Lamba (well vanilla, not Lamda@Edge) and CF solve my usual problems just with higher costs. CloudFlare workers actually let me do things I couldn't do. So the cost is still just as high as FaaS platforms BUT importantly I actually get something for it, something I can't otherwise have and important enough I am willing to pay the costs and overheads for.
Please advocate for real state-while-revalidate support internally! It would make tons of folks super happy.
I must admit I didn't go super deep on KV so I will come by discord at some point. I can share what we are doing and maybe it turns out KV is better suited.
This cycle will forever be this way. You only think that the new stuff is the best if this is your first rodeo. If it's not, it's just the right or wrong tradeoff for the point in time you are at.
I can't think of any other code of mine for which I'd use Beanstalk or something similar now, but it doesn't hurt to evaluate each project independently and pick the right tools for the job.
I told anyone that wasn't comfortable with Cloudformation + Packer and wanted to be on AWS to use it during that era. At that time things were pretty binary, you used PaaS like Beanstalk or Heroku or you were off in the wild west where there was no standards at all. People were doing immutable AMIs on ASGs at the high end and then everything down to pet AWS instances w/Capistrano or similar tools at the low end.
These days things are different, containers were a big deal. If you target k8s your application is pretty portable, can be run locally in something like kind to integration test the infra components, etc. You can take it to the 3 major clouds and bring most of your ingress, secrets, volume, etc configuration with you.
For me k8s has been a godsend because for once I haven't had to learn a whole new set of idioms each new place that are all equivalent but different in their own special snowflake ways.
How often have you seen someone actually do this, and was it ever actually as "easy" as they imagined? I feel like the advantages of "going with the flow" and using the stuff specific to the cloud environment you're in are substantial and I feel like the efforts to avoid vendor lock-in start to look insufficient once you try to actually execute most of the time.
My experience leads me to believe the main problems stem from adopting IaaS/PaaS specific stuff outside of k8s. If you aren't doing that they are very portable and even if you are if you are using standard stuff like managed PostgreSQL/MySQL then it's still easy.
Where that breaks down is very vendor specific stuff, ala Spanner, queues/logs like SQS/PubSub etc. If you are using those and your architecture is tied to specific semantics of them you are in for a rough time.
I personally try to avoid any and all of those. Not for vendor lock-in reasons but because I greatly dislike not being able to run everything locally in exactly the same configuration it will be deployed in.
So I run PostgreSQL in k8s using Patroni, Kafka or Pulsar for queue/log, etc and that is about it usually. I generally don't need/use other things. Maybe memcached if there is a good need for it over say Caffiene (in-memory cache).
This is awesome because it means it's super super easy to have staging/UAT environment or even preview environments on PRs and I can run integration tests against the whole stack E2E without spinning up/manipulating cloud provider resources.
So it was successful enough for long enough to get them to the point where someone cared about optimizing and paid you to come in, right? I don't see how you are disproving the article's argument that this is a great way to prototype ideas or get a first version of the product out there.
As long as it works and makes money it excuses shoddiness. Publishing a draft to fish for revenue is the reason we have so much poor software running out there. After it's in production, you will not be able to make a business case to re-make it right.
None of the tools that say they help you move faster matter after the first 3 months.
It's true they take less code and pieces to get started with. No k8s manifests, no Dockerfiles (usually) but if you don't hate your future self you will still use IaC to create the API Gateways/function contexts/DynamoDB/whatever, so you will still be dicking around with at least one of Terraform/Pulumi/similar.
The thing is the setup cost of a traditional lang + pg + k8s setup amortised over 3 months is essentially nothing and yet you have a setup that will last you effectively forever.
If your company wasn't going to die because of being a few days slower in that initial 3 month period you have came out ahead. You have a forever architecture and you avoided a whole bunch of costs and technical debt in the process.
Maybe the calculus is different if you don't have the same depth of infrastructure experience I do but for me it's obvious this is a faster and better way of delivering a prototype that gives it proper room to grow into production software.
It is true that _some_ resources don't have L2 constructs, but there's pretty damn good coverage of the most popular services - and, even when you have to drop back to L1, that's still better than hand-written CFN templates or abandoning IaC entirely!
[0] https://docs.aws.amazon.com/cdk/v2/guide/cdk_pipeline.html
If you're just on a single region/account then yes it's better than alternatives.
It is true that _some_ resources don't have L2 constructs, but there's pretty damn good coverage of the most popular services - and, even when you have to drop back to L1, that's still better than hand-written CFN templates or abandoning IaC entirely!
[0] https://docs.aws.amazon.com/cdk/v2/guide/cdk_pipeline.html
Then a lot of the resources depend on each other in funky ways, but they’re not always implicitly defined, so whether everything will be created in the right order is anyone’s guess and you often have to define the dependencies manually anyway.
Then when it fails to bring up/modify a stack you get the helpful message that resource x failed to create, but nearly never why.
Like, CDK is marginally better than plain CF, but something like Terraform or Pulumi is so much better that it’s hard to imagine anyone would want to use anything else.
Also, like someone else said, the stacks are directly bound to specific accounts/regions. When I bring domething up that is my QA environment, I’d enjoy it if I didn’t have to split every stack in twain (cloudfront certificates that can only be in us-east-1 or something).
Oh, and when you finally work through all these issues and work on your app for a few months, the dreaded message will come up… “Maximum of 500 resources in stack reached”.
Now enjoy splitting it all up.
It feels like you have a different boundary of what constitutes "infrastructure" than the CDK team do. I'd call both of those examples content _within_ an infrastructure component. Would you expect CDK to initialize an SQS Queue with a message on it? Or a DynamoDB table with data in it? I guess you could make that argument, but to me _populating_ your infrastructure is a very different task from _making_ it - it's an application-level task, not a infrastructure-level one.
Most of the rest of your comments are pretty subjective - I can't say I agree with most of them or have experienced them, but I also don't disbelieve you that they've happened to you.
Not sure what you mean by having an issue with "stacks [being] directly bound to specific accounts/regions", though. Why is that an issue? Why would you _want_ a stack that spans multiple account/regions? I just successfully created and validated a Certificate in us-west-2, so I'm not sure what your parenthetical means.
And on a large team there’s very little likelihood that everyone understands the different configuration layers. There’s very little likelihood that people understand the scope that changes to configuration might touch either. But when you work with CDK, it’s one tool to understand.
Faster and better way to grow into production ignores team level problems and team level constraints. At 10, 20, 100 engineers this is fine. But when you have dozens of teams you’re almost certainly losing efficiency by being prescriptive.
I find that this type of pedantry really surrounds the k8s ecosystem. I think k8s is an enterprise scale tool that has uses in enterprise applications. But the idea of suggesting it to everyone is lunacy
I’d consider it a massive, massive waste.
Large corpo spends 2M on a CRUD app and is happy with the result (e.g. consider it a success).
Another organisation (with more skilled people) builds exactly the same thing for just 50K, and also consider it a success.
The fact that it’s a success doesn’t mean that it wasn’t silly.
Sorry kind of lost track of the original thread here too.
Ergo you only see cases where it isn't working well.
Doesn't make it better than alternatives but still a valid point.
We do this in a handful of small services, none of which gets much traffic. The alternatives (e.g. app-engine, container in vm, or k8s) all require more infrastructure, would cost more (no scaling to zero), and are overall less suited for the kinds of problems we expect to see.
But I fully acknowledge this is not the ideal and optimal setup and we are paying more to AWS on the OpEx. However, we have gone 3 years without any downtime at full speed, and the fact that we were able to unlock net new revenue very quickly has made all project costs and OpEx very minuscule.
But I think once the application boundaries stabilize and we are able to take a breather from full on feature development, hope to migrate some of this to simpler containerized infrastructure.
To make this transition easier, we use Serverless-framework and decouple the app component architecture from deployment architecture, so developers don’t worry whether this is getting deployed to a lambda or container.
I make money by delivering features my customers care about.
We tried k8s, it was a ton of mental effort and time wasted when we could have been building more features. That particular bit of infrastructure is currently running in elastic beanstalk, where it's cheap and I don't have to think about it.
If Beanstalk + ELB + RDS or whatever is getting you by just fine then by all means. It's super easy to port that to k8s whenever you end up needing what it does, which might be never if your money/business scales well vs infra complexity.
Is this not just openly broadcasting your financial incentive to say “Whatever is popular is bad, pay me and I’ll make it good” ?
I have used (at large scale) GAE, Google Cloud Functions and Lambda, before that I also used all the previous "serverless-like" things including Beanstalk (anyone else even remember that?), Heroku and bunch of other PaaS things for various stuff.
It's all come and gone. To me serverless is just the latest in a long line of things that I get paid to replace with things that actually solve the problem and doesn't turn into a ball of spaghetti that no one is able to modify anymore.