News for hackers. Stuff that matters.
Re:So many bad takes. Learn from the Prime Video microservic... by waboremo on Sunday May 07, 2023 @01:47PM (#35853807)
Actually your comment is a great demonstration of poor engineering leadership. You've already decided on an outcome and are shifting the narrative to adhere to this outcome.

The labelling of JVM + K8 as boring, to posture why it's automatically superior to "serverless integration crap" is exactly the sort of thing poor engineering leadership does to avoid having the sort of difficult conversations necessary. There's no metrics involved, no studies, no prototypes, no financial assessments, none of that it's purely based on emotion.

You can envision the same exact thing happening (but to the preference of serverless) on the Prime Video team.

Re:So many bad takes. Learn from the Prime Video microservic... by mk89 on Sunday May 07, 2023 @02:38PM (#35854246)
> Actually your comment is a great demonstration of poor engineering leadership. You've already decided on an outcome and are shifting the narrative to adhere to this outcome.

For me it's actually the decision of a CTO of one of top 10 companies in the world in terms of software size/value/etc. to push it down to very smart people - that's bad leadership.

Why? Because I guess this thing didn't go as planned - a self fulfilling prophecy that shows how awesome serverless is. It backfired in a period where people are questioning whether all these megahypes are not actually just some marketing. Yes, he is responsible for that, for pushing it down to his people that "everything new must be started with serverless" first. When actually in some situations, as the article showed, a monolith or a (big) microservice would do the job better.

Why the hell hire super smart people when you have to tell them how to do things?

Re:So many bad takes. Learn from the Prime Video microservic... by hyperhopper on Sunday May 07, 2023 @02:51PM (#35854389)
> Why the hell hire super smart people when you have to tell them how to do things?

A very hard question. One side is that every smart engineer will do what's best for their use case.

The other is now you might have 100 smart engineers with 100 smart different solutions which can cause repeated work, integration issues, missed common solutions, etc.

Both approaches have lots of pros cons and tradeoffs

Re:So many bad takes. Learn from the Prime Video microservic... by jpgvm on Sunday May 07, 2023 @03:03PM (#35854498)
Generally the best approach there is to elevate a bunch of your smartest engineers into an architecture team that a) set good standards but also b) appreciates the nuance of problems sufficiently to know when to deviate.

A small team can probably only handle one tech stack, an org with 10 teams can probably handle 10 stacks if they needed to but can get efficiency by standardising on some number less than 10.

However making those calls shouldn't be coming down from the CTO, it should be made among the architects that actually design and build the systems.

Re:So many bad takes. Learn from the Prime Video microservic... by phil-m on Sunday May 07, 2023 @03:49PM (#35854967)
Where I'd consider a (good) CTO should be somehow responsible for the high-level architectural decisions. Accumulate enough expertise and know-how (via "architects" or rather very senior engineers) and then discuss the pros/cons and at the end weigh each argument and decide what may work best.
Re:So many bad takes. Learn from the Prime Video microservic... by j45 on Sunday May 07, 2023 @09:06PM (#35857059)
The smart have to design for the average of devs and not the smartest.

The smart design for easily creating beginners in thenfuture to work with what exists.

Too often smart devs have to be clever with coding, where clever architecture can often be more flexible.

Re:So many bad takes. Learn from the Prime Video microservic... by ec109685 on Sunday May 07, 2023 @04:06PM (#35855125)
Those smart engineers should be encouraged to devote their brainpower to other aspects of the business and technology stack.

Far too often teams use an opportunity to build something new as a science project to use all the latest wiz bang cloud features when the boring stack would work just fine.

For those that truly want to innovate on that aspect of the stack, they should be part of a central infrastructure team, whose job it is to build scalable solutions for the group.

Re:So many bad takes. Learn from the Prime Video microservic... by echelon on Sunday May 07, 2023 @09:39PM (#35857217)
You can hire anyone for postgres and k8s.

Not so much the fancy new hotness that will be weirdware in three years.

Re:So many bad takes. Learn from the Prime Video microservic... by robocat on Monday May 08, 2023 @05:21PM (#35867230)
Replying to your deleted comment on another thread:

IMHO if possible, grow your business without investors and retain control. Jason Cohen has an amazing talk on bootstrapping, with some very solid advice from his own experiences: https://m.youtube.com/watch?v=otbnC2zE2rw

Look at co-working spaces near you? Our business worked in an incubator for a bit, and the problems we saw with investors in other startups were eye-watering. We retained 100% ownership, and that was the right decision for us. Learn from the mistakes of others at different stages just by being around them.

Flick through my previous comments on VC for external links that explain my points better: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

I was a founder, and I don’t know shit about VC except through reading, so be appropriately skeptical about my thoughts and only take what is of value for your circumstances.

Good fortune.

Re:So many bad takes. Learn from the Prime Video microservic... by jpgvm on Sunday May 07, 2023 @01:51PM (#35853842)
It's not emotion. I make money from porting people off of poor technical choices onto solid ones. It comes from experience working with all these various systems.

I have used (at large scale) GAE, Google Cloud Functions and Lambda, before that I also used all the previous "serverless-like" things including Beanstalk (anyone else even remember that?), Heroku and bunch of other PaaS things for various stuff.

It's all come and gone. To me serverless is just the latest in a long line of things that I get paid to replace with things that actually solve the problem and doesn't turn into a ball of spaghetti that no one is able to modify anymore.

Re:So many bad takes. Learn from the Prime Video microservic... by Spivak on Sunday May 07, 2023 @02:39PM (#35854255)
Seconded, just got done with one of these. Literally 1200 separate lambdas, step functions, hundreds of kinesis streams, and sqs topics plus a smattering of ecs fargate tasks for when invariably something takes longer than the max lambda time. And oh god all the stupid supporting infrastructure to build layers.

There primary app was Rails, you know what replaced all of it? Running the synchronous tasks on the app servers themselves, the async ones on Sidekiq and changing the max instances on the two ASGs.

The only downside has been not being able to meme about how we're web scale anymore.

Re:So many bad takes. Learn from the Prime Video microservic... by jpgvm on Sunday May 07, 2023 @02:44PM (#35854326)
Not being able to make jokes about how fast it is to serve requests from /dev/zero must be absolutely killing your team.

However, great job. Once they get to that scale of objects it becomes really hard to clean it up - especially if most the people that wrote them have already left the company for example.

Curious if you ended up also saving money in the process?

Re:So many bad takes. Learn from the Prime Video microservic... by babyshake on Sunday May 07, 2023 @02:43PM (#35854312)
One potential reason to advocate for serverless architecture is it is relatively stateless and event-driven and therefore modular and easy to reason about.

Can you be more specific about how serverless tends to turn into a ball of spaghetti? It is simply because the lack of state becomes more of a bug than a feature?

Re:So many bad takes. Learn from the Prime Video microservic... by jpgvm on Sunday May 07, 2023 @02:57PM (#35854447)
State can be part of it. For example it can be way simpler to use a in-memory cache than a network attached cache and by doing so you save yourself a dependency, a potential network request that can fail, etc.

Also for event processing consider the case you need to process events in order for instance. That is a ton easier with stateful consumers, batching etc which while possible in serverless is usually more difficult/off the beaten path.

However by spaghetti I'm more referring to the architecture of your application being turned into networked components, i.e a distributed system.

It's pretty much accepted fact now that distributed systems are by far the most challenging area of both computer science and software engineering.

Serverless doesn't technically -mandate- this, you could put all your code in a giant function. However it strongly incentivises you not to. a) generally it's deployment times and cold start issues get worse with function size and b) it pushes you to adopt other "building blocks" which are themselves network attached.

Essentially it leads people into a massively distributed micro-service architecture right out of the gate. For most applications this is definitely the wrong choice even if you have the folks on staff that specialise in distributed systems, usually those folk are the last to advocate for anything distributed ironically.

Getting into the weeds as for why this is hard and all the problems you can run into is probably a bit much for a single HN comment but if you really are curious I suggest reading Leslie Lamport and John Ousterhout. Both have covered the problem space in depth with foundational papers, Leslies "Time, Clocks, and the Ordering of Events in a Distributed System" in particular is foundational.

Re:So many bad takes. Learn from the Prime Video microservic... by eternalban on Sunday May 07, 2023 @05:03PM (#35855569)
(quite impressed by the sheer number of comments you have made in this thread, btw)

> For example it can be way simpler to use a in-memory cache than a network attached cache and by doing so you save yourself a dependency, a potential network request that can fail, etc.

Elsewhere you decided against using cloudflare's kv because it was a poor fit for "cache invalidation" -- "way simpler"? -- and then of course turns out you really don't know if it's a bad fit.

> by spaghetti I'm more referring to the architecture of your application being turned into networked components, i.e a distributed system.

That is a novel definition of "spagetti". It's a pejorative term in software sense. Why?

> It's pretty much accepted fact now that distributed systems are by far the most challenging area of both computer science and software engineering.

Ahem. So it caching and cache invalidation. But that didn't stop you, so exactly how is this line you seem to draw with such vocal authority regarding "bad hard", "good hard" informed technically?

Re:So many bad takes. Learn from the Prime Video microservic... by _gabe_ on Sunday May 07, 2023 @05:26PM (#35855769)
> That is a novel definition of "spagetti". It's a pejorative term in software sense. Why?

I don't necessarily agree with everything OP is saying, and I certainly don't speak for them, but I completely understand what they mean by a big ball of microservice spaghetti. The spaghetti is distributed across a massive literal network of dependencies instead of being distributed off a giant graph of object dependencies, or goto statements or something. The spaghetti is real and just as messy in the microservice world as the monolith world. I should know, we inherited a massive one.

But this shouldn't be too much of a revelation since there is no silver bullet in taming software architecture. Architecting software is very subjective, and more often than not we make the wrong decisions and have to untangle the mistakes further down the road.

Re:So many bad takes. Learn from the Prime Video microservic... by eternalban on Sunday May 07, 2023 @05:59PM (#35856023)
> I completely understand what they mean by a big ball of microservice spaghetti

Sure, I understand that too, but that's not what was said. Here it is again:

> by spaghetti I'm more referring to the architecture of your application being turned into networked components, i.e a distributed system.

Re:So many bad takes. Learn from the Prime Video microservic... by Aeolun on Sunday May 07, 2023 @10:46PM (#35857558)
How is that not what was said. Your application is going to be spaghetti anyway, networked spaghetti is orders of magnitude worse.

I should know, I have a ball of networked spaghetti.

Re:So many bad takes. Learn from the Prime Video microservic... by intelVISA on Monday May 08, 2023 @05:35AM (#35859735)
Chef's law: spaghetti cannot be removed or added, only distributed.
Re:So many bad takes. Learn from the Prime Video microservic... by jpgvm on Sunday May 07, 2023 @06:24PM (#35856189)
> Elsewhere you decided against using cloudflare's kv because it was a poor fit for "cache invalidation" -- "way simpler"? -- and then of course turns out you really don't know if it's a bad fit.

I evaluated it at the time. I don't have the context loaded to remember exactly why it didn't work for my use-case but I'm going to look again because CFs platform moves pretty fast and perhaps it could be simpler than what we are currently doing.

> That is a novel definition of "spagetti". It's a pejorative term in software sense. Why?

Elsewhere it's been coined Lambda Pinball, perhaps you find that a more pleasing description?

> Ahem. So it caching and cache invalidation. But that didn't stop you, so exactly how is this line you seem to draw with such vocal authority regarding "bad hard", "good hard" informed technically?

Yes, both of which are generally necessary in distributed systems. i.e not only is it a hard problem in and of itself it requires solving sub-problems which are also some of the hardest problems.

I think you may have made my points for me.

Re:So many bad takes. Learn from the Prime Video microservic... by zerbinxx on Monday May 08, 2023 @12:17AM (#35858036)
I think in general you’re right on the money and have been throughout your comments here. Lambda pinball is a great term as well. I agree that serverless incentivizes a “nanoservice” approach that flies in the face of patterns like classes/factories or DDD, since everything is atomic you lose the concept of a service or domain boundary and everything can get muddled as hell.
Re:So many bad takes. Learn from the Prime Video microservic... by scarface74 on Sunday May 07, 2023 @05:59PM (#35856020)
> Also for event processing consider the case you need to process events in order for instance. That is a ton easier with stateful consumers, batching etc which while possible in serverless is usually more difficult/off the beaten path

By off the beaten path you mean a few lines of yaml to configure a FIFO SQS queue and have it trigger a Lambda?

Re:So many bad takes. Learn from the Prime Video microservic... by empthought on Sunday May 07, 2023 @09:25PM (#35857151)
I notice the GP hasn't responded to this. If Lambdas couldn't be a target for SNS or SQS, I wouldn't bother with them either. But that has been a core architectural pattern since 2018.

https://aws.amazon.com/blogs/aws/aws-lambda-adds-amazon-simp...

Re:So many bad takes. Learn from the Prime Video microservic... by ricardobeat on Sunday May 07, 2023 @03:37PM (#35854841)
Modular, maybe (though sometimes it's just introducing a network hop in between dependencies). Event-driven being easy to reason about? No way, never.
Re:So many bad takes. Learn from the Prime Video microservic... by erikpukinskis on Sunday May 07, 2023 @04:35PM (#35855389)
Yah, I was confused by that too. Out of:

1) event-driven

2) procedural

3) functional

4) inheritance

I would say event-driven is tied for last as the hardest to reason about.

Re:So many bad takes. Learn from the Prime Video microservic... by emodendroket on Sunday May 07, 2023 @04:39PM (#35855425)
I'd say it comes down to your choice of events. I also find these systems very easy to reason about.
Re:So many bad takes. Learn from the Prime Video microservic... by tharkun__ on Sunday May 07, 2023 @05:26PM (#35855767)
Instead of your IDE very easily telling you on click the callers or callees of things you'd rather have events where you have no idea who might be processing or sending a particular event in your system of hundreds of microservices?

Yeah, totally easy!

Re:So many bad takes. Learn from the Prime Video microservic... by ShroudedNight on Sunday May 07, 2023 @05:55PM (#35855995)
I think you're conflating problems here. IDLs allow interface definitions for interactions that cross compute and administrative boundaries, and event-driven architectures do not require compute distribution (Erlang/beam).

Different people have different strengths and anxieties. The suggestion that, with some regularity, the lack of (local) ambiguity of event message arrival may be of higher leverage relative to complexity than increasing the surface of an invokable interface does not seem beyond the pale.

Ultimately relinquishing a total temporal ordering is going to be painful regardless of whether it's local or distributed.

Re:So many bad takes. Learn from the Prime Video microservic... by tharkun__ on Sunday May 07, 2023 @08:02PM (#35856768)
That's fair enough if you use something that actually does this and also keeps track of users of your event. Not saying it is unsolvable.

What I've seen though is that in many cases it is unsolved for and it's very hard to reason about "if I remove this event because it's 5 versions too old already and I don't want to keep this backwards compatibility layer, will something "at the other end of our system I never even interact with" puke?

Granted some orgs are so big that it makes sense to solve for these large landscapes of services with backwards compatibility ad absurdum in order to allow anything to get done (been at that bank, done that). But reason about the entire system it does not make easier.

At many smaller orgs these services were also not built because of organizational issues that needed to be solved but because "everyone does this now" and there's no proper registry of properly defined interfaces and trying to find out if a change you were going to make is going to take you forever because you do need to version something or if you can just change things as needed is way harder than it's supposed to be at that size org.

Re:So many bad takes. Learn from the Prime Video microservic... by emodendroket on Monday May 08, 2023 @12:25AM (#35858074)
It's freeing you to reason about only one step rather than the entire system at all times.
Re:So many bad takes. Learn from the Prime Video microservic... by tharkun__ on Monday May 08, 2023 @08:24PM (#35868750)
Fair enough that it's freeing overall if your org is so large that it is faster to always be 100% backwards compatible for aeons.

I repeat: most orgs that go all in on this are actually so small that being so compatible is more of a drag than understanding the part of the system that is attached to the interface you are trying to change ;)

Re:So many bad takes. Learn from the Prime Video microservic... by emodendroket on Monday May 08, 2023 @10:03PM (#35869399)
Well that's what I mean about choosing the right events. If you are needing to keep multiple systems in your head all the time the boundaries are wrong. But it's true that evolving the interface of services is hard.
Re:So many bad takes. Learn from the Prime Video microservic... by xvinci on Sunday May 07, 2023 @04:04PM (#35855106)
Statelessnes is really a prerequisite for proper horizontal scaling (with small exceptions), so best practices for e.g. spring boot web applications will lead to a stateless architecture as well. Same goes for event-driven -> amqp, mqtt (and jms for java) plug easily into most modern web-frameworks, you can be as event-driven as you want to (which in my experience is an overhead to plain http / rest which needs to be justified)
Re:So many bad takes. Learn from the Prime Video microservic... by ShroudedNight on Sunday May 07, 2023 @05:37PM (#35855855)
> Statelessnes is really a prerequisite for proper horizontal scaling

I could understand making this claim w.r.t. Idempotence. Claiming statelessnes as a requirement strikes me as non-trivial enough to require some evidence or a citation thereto.

Re:So many bad takes. Learn from the Prime Video microservic... by namaria on Sunday May 07, 2023 @05:19PM (#35855705)
>stateless and event-driven and therefore modular

Non-sequitur

Re:So many bad takes. Learn from the Prime Video microservic... by plugin-baby on Sunday May 07, 2023 @03:05PM (#35854530)
Has GAE come and gone? Surely it’s still there, and capable of running the JVM? K8s seems like overkill for small teams which don’t already have experience
Re:So many bad takes. Learn from the Prime Video microservic... by jpgvm on Sunday May 07, 2023 @03:19PM (#35854675)
GAE is actually still pretty good tbh. However it has it's limitations and even though there are things like Cloud Run et al I still find GKE is the best option.
Re:So many bad takes. Learn from the Prime Video microservic... by vlovich123 on Sunday May 07, 2023 @03:27PM (#35854746)
Have you tried Cloudflare Workers? I work at Cloudflare but I’m genuinely curious what your feedback is on it as it does work quite differently from Lambda/Cloudflare functions.
Re:So many bad takes. Learn from the Prime Video microservic... by jpgvm on Sunday May 07, 2023 @03:37PM (#35854850)
Yes actually, currently using in production. I actually advocated for the move to Cloudflare and we ended up adopting workers because Cloudflare unfortunately has broken stale-while-revalidate support. So I built an implementation in Workers that abuses the Cache API to store our own TTLs and also our revalidating state. It works pretty well as we don't end up with too many isolates per POP but it's not perfect.

I didn't end up using KV or other features yet, they weren't a good fit for the cache revalidation implementation.

I think at-least for now I see it differently to FaaS services like Lambda/CloudFunctions because I don't intend to use it to implement application logic, rather just augment (or in this case shore up deficiencies) in the edge.

We may consider doing more stuff at the edge later for latency advantages. This to me changes the calculus because Lamba (well vanilla, not Lamda@Edge) and CF solve my usual problems just with higher costs. CloudFlare workers actually let me do things I couldn't do. So the cost is still just as high as FaaS platforms BUT importantly I actually get something for it, something I can't otherwise have and important enough I am willing to pay the costs and overheads for.

Please advocate for real state-while-revalidate support internally! It would make tons of folks super happy.

Re:So many bad takes. Learn from the Prime Video microservic... by vlovich123 on Sunday May 07, 2023 @03:49PM (#35854966)
Interesting. Since I TL KV currently I’m particularly interested in the ways it’s not working for you. We’re in the process of rolling out major changes here that might help you here so hit me up on the KV Discord or my username at cloudflare.com. I actually do have a meeting with the Cache team on Monday about something related and I’ll ask about it if you could message me what specifically is broke / what the semantics you want are (I’m not intimately familiar with all the HTTP semantics).
Re:So many bad takes. Learn from the Prime Video microservic... by jpgvm on Sunday May 07, 2023 @04:09PM (#35855157)
Shot you an email.

I must admit I didn't go super deep on KV so I will come by discord at some point. I can share what we are doing and maybe it turns out KV is better suited.

Re:So many bad takes. Learn from the Prime Video microservic... by vlovich123 on Sunday May 07, 2023 @07:17PM (#35856510)
Grrrt autocomplete. Lambda/CloudFunctions.
Re:So many bad takes. Learn from the Prime Video microservic... by grogenaut on Sunday May 07, 2023 @03:37PM (#35854839)
I remember beanstalk. We used it for years at twitch. We got a shit ton of mileage out of it. It's easy to talk shit about the problems of the old tech, and beanstalk has a lot of issues. But it also let us build a ton of stuff much easier than the alternatives.

This cycle will forever be this way. You only think that the new stuff is the best if this is your first rodeo. If it's not, it's just the right or wrong tradeoff for the point in time you are at.

Re:So many bad takes. Learn from the Prime Video microservic... by noduerme on Sunday May 07, 2023 @04:22PM (#35855279)
As a one man shop, I made a decision years ago to try Beanstalk for a particular app backend / API. It's the only "serverless" deployment I've done. It still works fine for that purpose, and it's less trouble than manually deploying to multiple servers. Forced updates are an occasional headache but at least they're all in one place. The code does a lot of serverside computation for a lot of client connections, so that part needs to scale, and since there's relatively little database stuff it just connects to a single RDS instance.

I can't think of any other code of mine for which I'd use Beanstalk or something similar now, but it doesn't hurt to evaluate each project independently and pick the right tools for the job.

Re:So many bad takes. Learn from the Prime Video microservic... by jpgvm on Sunday May 07, 2023 @04:31PM (#35855354)
Yeah Beanstalk was pretty simple and decent for it's time.

I told anyone that wasn't comfortable with Cloudformation + Packer and wanted to be on AWS to use it during that era. At that time things were pretty binary, you used PaaS like Beanstalk or Heroku or you were off in the wild west where there was no standards at all. People were doing immutable AMIs on ASGs at the high end and then everything down to pet AWS instances w/Capistrano or similar tools at the low end.

These days things are different, containers were a big deal. If you target k8s your application is pretty portable, can be run locally in something like kind to integration test the infra components, etc. You can take it to the 3 major clouds and bring most of your ingress, secrets, volume, etc configuration with you.

For me k8s has been a godsend because for once I haven't had to learn a whole new set of idioms each new place that are all equivalent but different in their own special snowflake ways.

Re:So many bad takes. Learn from the Prime Video microservic... by emodendroket on Sunday May 07, 2023 @04:36PM (#35855391)
> You can take it to the 3 major clouds and bring most of your ingress, secrets, volume, etc configuration with you.

How often have you seen someone actually do this, and was it ever actually as "easy" as they imagined? I feel like the advantages of "going with the flow" and using the stuff specific to the cloud environment you're in are substantial and I feel like the efforts to avoid vendor lock-in start to look insufficient once you try to actually execute most of the time.

Re:So many bad takes. Learn from the Prime Video microservic... by jpgvm on Sunday May 07, 2023 @05:07PM (#35855598)
I have done one massive AWS + kops > GKE migration and several smaller migrations between AWS/GCP/Azure.

My experience leads me to believe the main problems stem from adopting IaaS/PaaS specific stuff outside of k8s. If you aren't doing that they are very portable and even if you are if you are using standard stuff like managed PostgreSQL/MySQL then it's still easy.

Where that breaks down is very vendor specific stuff, ala Spanner, queues/logs like SQS/PubSub etc. If you are using those and your architecture is tied to specific semantics of them you are in for a rough time.

I personally try to avoid any and all of those. Not for vendor lock-in reasons but because I greatly dislike not being able to run everything locally in exactly the same configuration it will be deployed in.

So I run PostgreSQL in k8s using Patroni, Kafka or Pulsar for queue/log, etc and that is about it usually. I generally don't need/use other things. Maybe memcached if there is a good need for it over say Caffiene (in-memory cache).

This is awesome because it means it's super super easy to have staging/UAT environment or even preview environments on PRs and I can run integration tests against the whole stack E2E without spinning up/manipulating cloud provider resources.

Re:So many bad takes. Learn from the Prime Video microservic... by noduerme on Sunday May 07, 2023 @10:15PM (#35857391)
One nice thing about Beanstalk was that you didn't (don't) have to deal with containers on the toolchain at all. You basically zip the directory structure you'd normally serve and it just gets rolled out to all the instances. What I figured out early (the hard way) was that it wasn't worth the time trying to screw around with leader instances that installed and ran cron jobs or anything like that. I just have an little separate ec2 instance that acts as a heartbeat for cleaning up the DB and buckets, rather than trying to get Beanstalk to do that consistently. I think it's extremely well suited for one layer, the API; and counterintuitively I find it easier not to containerize that, because it gives me more flexibility to switch DBs or storage engines without touching the API, and without having to rework a container and redeploy the API itself.
Re:So many bad takes. Learn from the Prime Video microservic... by emodendroket on Sunday May 07, 2023 @04:37PM (#35855404)
> I make money from porting people off of poor technical choices onto solid ones

So it was successful enough for long enough to get them to the point where someone cared about optimizing and paid you to come in, right? I don't see how you are disproving the article's argument that this is a great way to prototype ideas or get a first version of the product out there.

Re:So many bad takes. Learn from the Prime Video microservic... by rat9988 on Sunday May 07, 2023 @04:52PM (#35855508)
"So it was successful enough for long enough " Which doesn't necessarily make it a good choice at any point. It just says it wasn't enough of a bad technical choice to make the product sink.
Re:So many bad takes. Learn from the Prime Video microservic... by emodendroket on Sunday May 07, 2023 @05:01PM (#35855557)
OK, sure, it's not ironclad proof that it was the right decision. But it's certainly not proof it was the wrong decision when it was made.
Re:So many bad takes. Learn from the Prime Video microservic... by namaria on Sunday May 07, 2023 @05:18PM (#35855695)
"It works and makes money" is at the root of so much shoddy workmanship in software I think you're making a different point than what you're trying to.
Re:So many bad takes. Learn from the Prime Video microservic... by emodendroket on Sunday May 07, 2023 @11:53PM (#35857894)
The argument of TFA is that starting with serverless and evolving away from it bit by bit as it is revealed to be necessary is a sound strategy most of the time. Whatever you may think of that argument, “I have been hired to optimize projects built using serverless and get them off of it” is not a rebuttal.
Re:So many bad takes. Learn from the Prime Video microservic... by namaria on Monday May 08, 2023 @02:23AM (#35858700)
Thanks for rehashing how this thread got started for me?
Re:So many bad takes. Learn from the Prime Video microservic... by emodendroket on Monday May 08, 2023 @03:10AM (#35858926)
What is this different point you think I’m making then?
Re:So many bad takes. Learn from the Prime Video microservic... by namaria on Monday May 08, 2023 @05:36AM (#35859746)
>this is a great way to prototype ideas or get a first version of the product out there

As long as it works and makes money it excuses shoddiness. Publishing a draft to fish for revenue is the reason we have so much poor software running out there. After it's in production, you will not be able to make a business case to re-make it right.

Re:So many bad takes. Learn from the Prime Video microservic... by emodendroket on Monday May 08, 2023 @10:55AM (#35862217)
Apparently you will since the other guy in this thread has made this a specialization.
Re:So many bad takes. Learn from the Prime Video microservic... by re-thc on Monday May 08, 2023 @12:13AM (#35858004)
This has nothing to do with software. Same applies for things like buildings where the developers get away with so many defects even after people pay huge $$$ for it.
Re:So many bad takes. Learn from the Prime Video microservic... by jpgvm on Sunday May 07, 2023 @04:52PM (#35855512)
My opinion from doing both these ports and greenfield is this:

None of the tools that say they help you move faster matter after the first 3 months.

It's true they take less code and pieces to get started with. No k8s manifests, no Dockerfiles (usually) but if you don't hate your future self you will still use IaC to create the API Gateways/function contexts/DynamoDB/whatever, so you will still be dicking around with at least one of Terraform/Pulumi/similar.

The thing is the setup cost of a traditional lang + pg + k8s setup amortised over 3 months is essentially nothing and yet you have a setup that will last you effectively forever.

If your company wasn't going to die because of being a few days slower in that initial 3 month period you have came out ahead. You have a forever architecture and you avoided a whole bunch of costs and technical debt in the process.

Maybe the calculus is different if you don't have the same depth of infrastructure experience I do but for me it's obvious this is a faster and better way of delivering a prototype that gives it proper room to grow into production software.

Re:So many bad takes. Learn from the Prime Video microservic... by emodendroket on Sunday May 07, 2023 @05:05PM (#35855584)
I am more of a generalist than you are but I have worked with a Kubernetes setup and I've also worked with going all-in on Lambda and other AWS services managed by CDK and I don't share your intuition. Besides the initial setup, the former also has other ongoing maintenance things that are harder to deal with. It does give you more flexibility if whatever you're doing isn't a good fit for those tools, though (though obviously nothing stops you from provisioning EC2 instances or whatever through other tools).
Re:So many bad takes. Learn from the Prime Video microservic... by Aeolun on Sunday May 07, 2023 @10:38PM (#35857505)
I have a hard time imagining anyone doing anything non trivial with CDK and walking away happy.
Re:So many bad takes. Learn from the Prime Video microservic... by emodendroket on Sunday May 07, 2023 @11:51PM (#35857882)
Why? Do you have something specific to say about why you dislike it? I think it’s pretty good.
Re:So many bad takes. Learn from the Prime Video microservic... by scubbo on Monday May 08, 2023 @12:21AM (#35858055)
Oddly I'm not able to reply directly to re-thc, so I hope you don't mind me commenting here that - I suspect that they're not deploying with a CDK-defined Code Pipeline (starter article[0], in-depth example[1]), which makes cross account/region deployments so ludicrously smooth and simple that it often has newbies refuse to believe that it can have worked _that_ easily.

It is true that _some_ resources don't have L2 constructs, but there's pretty damn good coverage of the most popular services - and, even when you have to drop back to L1, that's still better than hand-written CFN templates or abandoning IaC entirely!

[0] https://docs.aws.amazon.com/cdk/v2/guide/cdk_pipeline.html

[1] https://pipelines.devops.aws.dev/

Re:So many bad takes. Learn from the Prime Video microservic... by emodendroket on Monday May 08, 2023 @12:23AM (#35858067)
If the reply link doesn't appear you can still reply by clicking on the timestamp. I don't know why it works that way.
Re:So many bad takes. Learn from the Prime Video microservic... by scubbo on Thursday May 11, 2023 @01:23AM (#35897920)
Odd - it's showing up now. I'll remember for next time. Thanks!
Re:So many bad takes. Learn from the Prime Video microservic... by re-thc on Monday May 08, 2023 @12:11AM (#35857990)
Cross region/account usage is terrible. It's also 1/2 baked. Some things have higher level constructs but some don't. Lots of things have to be "escaped" to get it to work.

If you're just on a single region/account then yes it's better than alternatives.

Re:So many bad takes. Learn from the Prime Video microservic... by scubbo on Thursday May 11, 2023 @01:23AM (#35897917)
I suspect that you're not deploying with a CDK-defined Code Pipeline (starter article[0], in-depth example[1]), which makes cross account/region deployments so ludicrously smooth and simple that it often has newbies refuse to believe that it can have worked _that_ easily.

It is true that _some_ resources don't have L2 constructs, but there's pretty damn good coverage of the most popular services - and, even when you have to drop back to L1, that's still better than hand-written CFN templates or abandoning IaC entirely!

[0] https://docs.aws.amazon.com/cdk/v2/guide/cdk_pipeline.html

[1] https://pipelines.devops.aws.dev/

Re:So many bad takes. Learn from the Prime Video microservic... by Aeolun on Monday May 08, 2023 @12:26PM (#35863514)
It’s just… half baked. It feels like the CDK team constantly has to work circles around the lacking functionality in CFN. Want to upload a docker image to ECR, use this hacky lambda function. Want to create an object in an S3 bucket? Use this other hacky lambda function.

Then a lot of the resources depend on each other in funky ways, but they’re not always implicitly defined, so whether everything will be created in the right order is anyone’s guess and you often have to define the dependencies manually anyway.

Then when it fails to bring up/modify a stack you get the helpful message that resource x failed to create, but nearly never why.

Like, CDK is marginally better than plain CF, but something like Terraform or Pulumi is so much better that it’s hard to imagine anyone would want to use anything else.

Also, like someone else said, the stacks are directly bound to specific accounts/regions. When I bring domething up that is my QA environment, I’d enjoy it if I didn’t have to split every stack in twain (cloudfront certificates that can only be in us-east-1 or something).

Oh, and when you finally work through all these issues and work on your app for a few months, the dreaded message will come up… “Maximum of 500 resources in stack reached”.

Now enjoy splitting it all up.

Re:So many bad takes. Learn from the Prime Video microservic... by scubbo on Thursday May 11, 2023 @01:34AM (#35898004)
> It feels like the CDK team constantly has to work circles around the lacking functionality in CFN. Want to upload a docker image to ECR, use this hacky lambda function. Want to create an object in an S3 bucket? Use this other hacky lambda function.

It feels like you have a different boundary of what constitutes "infrastructure" than the CDK team do. I'd call both of those examples content _within_ an infrastructure component. Would you expect CDK to initialize an SQS Queue with a message on it? Or a DynamoDB table with data in it? I guess you could make that argument, but to me _populating_ your infrastructure is a very different task from _making_ it - it's an application-level task, not a infrastructure-level one.

Most of the rest of your comments are pretty subjective - I can't say I agree with most of them or have experienced them, but I also don't disbelieve you that they've happened to you.

Not sure what you mean by having an issue with "stacks [being] directly bound to specific accounts/regions", though. Why is that an issue? Why would you _want_ a stack that spans multiple account/regions? I just successfully created and validated a Certificate in us-west-2, so I'm not sure what your parenthetical means.

Re:So many bad takes. Learn from the Prime Video microservic... by nostrebored on Sunday May 07, 2023 @05:30PM (#35855785)
The IAC piece is the same across both. Except with k8s you’re managing infrastructure across cloud, k8s, potentially frameworks like helm, etc.

And on a large team there’s very little likelihood that everyone understands the different configuration layers. There’s very little likelihood that people understand the scope that changes to configuration might touch either. But when you work with CDK, it’s one tool to understand.

Faster and better way to grow into production ignores team level problems and team level constraints. At 10, 20, 100 engineers this is fine. But when you have dozens of teams you’re almost certainly losing efficiency by being prescriptive.

I find that this type of pedantry really surrounds the k8s ecosystem. I think k8s is an enterprise scale tool that has uses in enterprise applications. But the idea of suggesting it to everyone is lunacy

Re:So many bad takes. Learn from the Prime Video microservic... by Aeolun on Sunday May 07, 2023 @10:26PM (#35857436)
If you consider building your service that should take 3000 man hours to build in 300k instead, overspending by a factor if a 100, and then still succeeding, then yeah, you could consider that a success.

I’d consider it a massive, massive waste.

Re:So many bad takes. Learn from the Prime Video microservic... by emodendroket on Sunday May 07, 2023 @11:50PM (#35857870)
I have no idea what you’re arguing with this one.
Re:So many bad takes. Learn from the Prime Video microservic... by Aeolun on Monday May 08, 2023 @12:37PM (#35863699)
You can build exactly the same application even if you throw different amounts of money at it.

Large corpo spends 2M on a CRUD app and is happy with the result (e.g. consider it a success).

Another organisation (with more skilled people) builds exactly the same thing for just 50K, and also consider it a success.

The fact that it’s a success doesn’t mean that it wasn’t silly.

Sorry kind of lost track of the original thread here too.

Re:So many bad takes. Learn from the Prime Video microservic... by trollied on Sunday May 07, 2023 @04:49PM (#35855480)
Amen, brother. Love you.
Re:So many bad takes. Learn from the Prime Video microservic... by unusualmonkey on Sunday May 07, 2023 @06:16PM (#35856137)
Bit of survivorship bias there! No-one is going to hire you to port off of serverless, if they're happy with it.

Ergo you only see cases where it isn't working well.

Re:So many bad takes. Learn from the Prime Video microservic... by jpgvm on Sunday May 07, 2023 @06:18PM (#35856147)
Yeah that is fair. There is probably serverless chugging away fine in lots of places.

Doesn't make it better than alternatives but still a valid point.

Re:So many bad takes. Learn from the Prime Video microservic... by unusualmonkey on Sunday May 07, 2023 @06:52PM (#35856369)
I use serverless successfully - handing tens of millions of annual revenue.

We do this in a handful of small services, none of which gets much traffic. The alternatives (e.g. app-engine, container in vm, or k8s) all require more infrastructure, would cost more (no scaling to zero), and are overall less suited for the kinds of problems we expect to see.

Re:So many bad takes. Learn from the Prime Video microservic... by slrainka on Monday May 08, 2023 @01:36AM (#35858474)
I too have a small team that built a greenfield application that generates about $100M annually for a large hotel chain. Serverless allowed us to get to market fast and everyone in the team is a feature developer — 0 infra folks. I think that’s the biggest value we have been able to derive from this compared all the other internal App teams who struggle with release cadences, experimentation and TTV.

But I fully acknowledge this is not the ideal and optimal setup and we are paying more to AWS on the OpEx. However, we have gone 3 years without any downtime at full speed, and the fact that we were able to unlock net new revenue very quickly has made all project costs and OpEx very minuscule.

But I think once the application boundaries stabilize and we are able to take a breather from full on feature development, hope to migrate some of this to simpler containerized infrastructure.

To make this transition easier, we use Serverless-framework and decouple the app component architecture from deployment architecture, so developers don’t worry whether this is getting deployed to a lambda or container.

Re:So many bad takes. Learn from the Prime Video microservic... by stickfigure on Sunday May 07, 2023 @06:52PM (#35856372)
> I make money from porting people off of poor technical choices

I make money by delivering features my customers care about.

We tried k8s, it was a ton of mental effort and time wasted when we could have been building more features. That particular bit of infrastructure is currently running in elastic beanstalk, where it's cheap and I don't have to think about it.

Re:So many bad takes. Learn from the Prime Video microservic... by jpgvm on Sunday May 07, 2023 @08:00PM (#35856759)
I actually think Beanstalk/Cloud Run/friends are fine. I should have specified that Functions as a Service is the main mess I think people should be avoiding.

If Beanstalk + ELB + RDS or whatever is getting you by just fine then by all means. It's super easy to port that to k8s whenever you end up needing what it does, which might be never if your money/business scales well vs infra complexity.

Re:So many bad takes. Learn from the Prime Video microservic... by alex_lav on Sunday May 07, 2023 @10:03PM (#35857325)
> I make money from porting people off of poor technical choices onto solid ones.

Is this not just openly broadcasting your financial incentive to say “Whatever is popular is bad, pay me and I’ll make it good” ?

Re:So many bad takes. Learn from the Prime Video microservic... by AndrewKemendo on Sunday May 07, 2023 @03:32PM (#35854796)
Fully agree with this - it completely ignores the context and domain for a one size fits all solution, which is precisely what is wrong with engineering today - lack of holism.

The OP's comment "Also generally the conception that building things with containers + k8s actually takes more time than serverless is a horseshit take IMO. It doesn't and if it's taking you longer you are probably doing something wrong."

I couldn't disagree with more from a hands on practical perspective. That will 100% take anyone longer to spin up for an MVP (for more cost) than spinning up a FreeBSD server and I'm unaware of anything that needs more than that for an MVP.

This isn't even mentioning the fact that every second your data sits in an AWS/CGP/Azure datacenter - you are not only telling them what kind of data you have (don't pretend like the hosts don't know and can't tell) but you're increasing your

Re:So many bad takes. Learn from the Prime Video microservic... by maccard on Sunday May 07, 2023 @03:48PM (#35854964)
> I couldn't disagree with more from a hands on practical perspective. That will 100% take anyone longer to spin up for an MVP (for more cost) than spinning up a FreeBSD server and I'm unaware of anything that needs more than that for an MVP.

I agree with OP. If you hand me a binary and instructions on how to run it, I can have it running on either AWS or DO in a container, connected to a managed database with snapshots enabled in about 30 minutes. If the toolchain is easy to set up, I can have CI/CD with 0-downtime deployments with auto-rollback in another 30 minutes, and you won't need to touch either of those until you're hitting thousands of concurrent users for a typical web app.

> you are not only telling them what kind of data you have (don't pretend like the hosts don't know and can't tell)

this is FUD of the n'th degree - if you suspect AWS are spying on your managed RDS data, I don't know what to tell you.

Re:So many bad takes. Learn from the Prime Video microservic... by AndrewKemendo on Sunday May 07, 2023 @04:50PM (#35855489)
Great…So you quickly put yourself in a position where the longer you’re on the service the more expensive it will be to migrate

And yes Amazon uses AWS utilization to keep track of promising startups or places that they want signal about market changes.

Re:So many bad takes. Learn from the Prime Video microservic... by maccard on Sunday May 07, 2023 @05:12PM (#35855637)
> Great…So you quickly put yourself in a position where the longer you’re on the service the more expensive it will be to migrate

With 30 minutes work, I've set up a production quality solution that I won't have to touch for months/years, and has almost no vendor lock-in. Containers are portable, you can run them on AWS, DO, GCP, or install docker on a bare metal instance, it's up to you. Until you _do_ decide to migrate, it's pretty much 0 upkeep (I checked one of the work projects I stood up and we haven't touched the infrastructure side of it for 9 months).

> And yes Amazon uses AWS utilization to keep track of promising startups or places that they want signal about market changes.

That's not what you said at first, you said "you are not only telling them what kind of data you have (don't pretend like the hosts don't know and can't tell)". You're telling AWS that you're running containers and a database.

Re:So many bad takes. Learn from the Prime Video microservic... by scarface74 on Sunday May 07, 2023 @06:04PM (#35856055)
Yes, and I’m sure after you set things up with “no vendor lock in”, you came back years later when the company grew and organized a large migration where you had to deal with budgets, transferring data, regression testing, going through architecture review boards, the PMO, permissions, compliance, training, etc.

I’m also sure after many years no vendor specific corner cases snuck in there you didn’t realize and you did all of that with no regressions.

Re:So many bad takes. Learn from the Prime Video microservic... by echelon on Sunday May 07, 2023 @03:45PM (#35854935)
> poor engineering leadership.

Or fantastic engineering leadership.

OP has constrained the technology choices of the org into a very capable set of primitives. The entire org can move faster by sitting atop the same rails.

K8s + tried and true tech lets the org spin up additional compute anywhere and easily migrate off a particular vendor. Lift and shift is work, but not nearly as bad when the rails are simple.

Re:So many bad takes. Learn from the Prime Video microservic... by jpgvm on Sunday May 07, 2023 @04:12PM (#35855181)
Having done the dreaded AWS + kops -> GKE migration I must agree, it was hard but nowhere near as hard as it could have been.
Re:So many bad takes. Learn from the Prime Video microservic... by nostrebored on Sunday May 07, 2023 @04:45PM (#35855465)
> simple

> k8s

Pick one

Re:So many bad takes. Learn from the Prime Video microservic... by jpgvm on Sunday May 07, 2023 @05:01PM (#35855556)
Eh. There are different types of complexity.

k8s has a nice contained local complexity that I prefer.

It's not simple internally but it's simple to use if you know how it works.

So really it's not "k8s isn't simple" it's more "nothing is simple" so pick your preferred type of complexity.

Re:So many bad takes. Learn from the Prime Video microservic... by scarface74 on Sunday May 07, 2023 @09:48PM (#35857262)
> It's not simple internally but it's simple to use if you know how it works.

Isn’t that literally true about anything?

Re:So many bad takes. Learn from the Prime Video microservic... by stcroixx on Sunday May 07, 2023 @04:09PM (#35855159)
I agree with him that serverless should not be the default and there should be a clear justification for needing those features.

He then fails to apply the same rule to k8. Almost nobody needs that either and treating it as the default not a good strategy.

Re:So many bad takes. Learn from the Prime Video microservic... by Aperocky on Sunday May 07, 2023 @10:03PM (#35857324)
Exactly, not everything needs scale.

For a low traffic, low use system, lambda + dynamo might just be the cheapest and easiest to implement and maintain.

And there are generally more of those type of systems.

Re:So many bad takes. Learn from the Prime Video microservic... by Aeolun on Sunday May 07, 2023 @10:56PM (#35857614)
Depends? Is your load consistent and latency sensitive? Then having a small ECS instance may be a lot better.
Re:So many bad takes. Learn from the Prime Video microservic... by jasonm23 on Monday May 08, 2023 @12:20AM (#35858051)
Yes, to call Serverless hype train and then talk of K8s as the "boring solid choice" is pretty funny.
Re:So many bad takes. Learn from the Prime Video microservic... by Aeolun on Sunday May 07, 2023 @10:23PM (#35857424)
> it's purely based on emotion

Maybe charitably assume it’s based on years of experience doing this exact thing?

It’s certainly my experience.