> It’s crazy to me to think someone actually thought using step functions and lambdas in this way was a good idea.

Maybe Amazon used this to test step functions and lambdas.

I think there's a lot of projects within Amazon that get extra marks for dogfooding AWS and for providing materials / whitepapers for use cases of various AWS services.

I've done some AWS training, especially the introductory ones are intended to get you to use as many different services as possible. I'm sure Amazon's internal training is just that.

> I've done some AWS training, especially the introductory ones are intended to get you to use as many different services as possible

That sounds like an obvious good thing right? The training isn't supposed to teach you how to solve problems, its supposed to teach you how to use a particular set of tools.

Ish? If we are talking about "hammer training," then I can see the argument. Teach me about the different hammers and why I would pay attention to the weight and such.

But all too often, the training picks something like "build a clubhouse." And in that, I would expect legit solutions based on specifications of a clubhouse that you are wanting to build.

Language advocates fall hard on this. They will "build a TODO application" to show off part of the language. But end up with an application that can't do what folks would want a TODO application to actually do. It is very frustrating.

I once worked on a payment processing system. Files came in with about a thousand payments, in a textual format, 10-20 short lines of text each.

The geniuses who led the project before me decided that the first step was to split the file into single payment chunks, then put each one into RabbitMQ, from where a different service would pick them up and parse them - then encode the parsed payments in JSON, and put them back in RabbitMQ, for a third process to pick up and handle. This was all for performance, of course.

I tried to explain that just parsing the whole file in one go would be much faster than splitting it up, writing it to a queue, and reading it back in. That it probably wouldn't be much slower than the splitting. But nobody was interested. Obviously distributing the work would be much faster.

It was a weird experience. My colleagues just had no ability to reason about the "physics" of what computers did. I think they mostly had Rails backgrounds, and I wonder if you just don't exercise that muscle much doing Rails.

That intuitive feel of "what's faster" has been lost ever since computing power exploded. Back when the resources were at a premium, we were just better engineers. Not all of us had the command of Leetcode algorithm problems, but we rarely missed a database index and knew to stay close to the metal as much as possible, avoiding network trips, avoiding database trips.

It still holds true today, and software has become bloated and slower because these simple rules are being ignored. It's all in the cloud, it's all distributed, it's all "free".

As for Rails - it's the same problem but it's different. If you stay too high up in the abstraction orbit, it's hard to think of the nitty gritty of performance.

What is the cloud doing? What is the ORM doing? Once it's all "magic", you just keep piling on non-performant crap until it becomes an emergency. Then you just add more cache :)

I don't think it's necessarily abstraction, it's just complexity. We've been given a thousand footguns and we're expected to know how to use each one. You didn't consider microservices and lambda twenty years ago because they didn't exist. But you also didn't have to build an app that worked everywhere and looked good on every screen.

There’s also the fact that we shouldn’t be thinking about problems the way they used to be solved - not that we shouldn’t be considering database calls and network, but because compute/memory/storage are cheaper than they used to be, we’re able to make more tradeoffs to save developer time or to improve experience. Back when the option was to further optimize how many registers your assembly is using or to leave an application in a virtually unusable state, you didn’t really have a choice. You optimized the small details

Oh, compute and storage are cheap alright, until they are not.

At many jobs I had to deal with the all-hands-on-deck, code red emergency "our AWS bill got too damn high, what can we cut" situation. I can guarantee you a ton of companies are going through this exercise right now.

Some software engineers purposefully develop using fewer resources to force themselves to reason about efficiency.

Abundance leads to complacency and laziness, and those resources are no longer cheap if you are abusing them.

Since many have been told repeatedly that rails is slow, maybe some just stopped caring if the code actually got slow.

I saw this exact same pattern before. I think it's actually common, because I've also heard people in favor of micro services point to this design as a point in their favor.

One reason always given is the ability to replay chunks if the system goes down, at least for Kafka.

At one level that SORT of makes sense, but this extreme resilience becomes so complex that I can't imagine a simpler "retry where you were last time" logic cannot replace that. It's not like that problem has not been solved thousands of times before these new cool ways were the new expensive hotness.

Also, I doubt many of these places even know if their system is that tested for resilience. Do they know they are less likely to lose more data than that one record, which may not even be that precious?

> It was a weird experience. My colleagues just had no ability to reason about the "physics" of what computers did. I think they mostly had Rails backgrounds, and I wonder if you just don't exercise that muscle much doing Rails.

Same experience here. I feel like I'm always repeating myself when explaining to other developers why we should avoid moving data around unnecessary and prefer to operate directly in the database when necessary, specially when it comes to large data migrations. Also important is why it is bad to do a bunch of small operations instead of batching them in chunks closer to the page size (8 kB).

It sounds like somebody read that RabitMQ was the hot new thing, and then came up with a design to justify using it.

>My colleagues just had no ability to reason about the "physics" of what computers did. I think they mostly had Rails backgrounds,

Not surprised. Rails seems to encourage the worst behavior in developers.

I don't think that's on Rails, more that people with less experience tend to use more indirection in their solutions because each step of the problem is seen as independent and breaking everything down into its own class or method enhances 'readability'.

- Step one: put file in queue

- Step two: break down file and insert rows

- Step three: convert to json

- Step four: compute results

- etc...

Once your experience improves you start to understand the cost of such indirection and realise that the problem could have been solved in a much more straightforward manner.

That would be pretty stupid to do with Rails, there's a pretty high cost to starting jobs, so it makes more sense to batch everything.

No True Microservice would do something that dumb.

This explains why Prime Video is the only streaming service that can never stay stable on my gigabit connection, ugh. Sometimes it drops resolution until it's 360p.

This is true.

> Splitting thousands of hours of video into 1-2s chunks, uploading them to s3

Eh, that's a separate thing - that's how modern streaming works, i.e. it's what allowed streaming to become really, really good, and it's how pretty much every major streaming platform works today.

Serving an HLS segmented file is not the same as doing the processing work that enables a good transcode to that HLS.

Sorry, who is saying those two are the same?

I don't know about 1-2s chunks, but have you ever had to process a UHD HDR ProRes master with 24 channels of audio? Processing that on a single machine (depending on the level of processing) can easily take up to 10 hours. I definitely agree the 1-2s chunks makes no sense, but cutting it up into segments is the only way to get realistically acceptable processing times.

> It’s crazy to me to think someone actually thought using step functions and lambdas in this way was a good idea.

It’s only crazy to you because you don’t know what you’re doing. But it seems the average HNer isn’t smart enough to digest anything beyond “AWS” and “Cost” before raging about stuff they know nothing about.

> Splitting thousands of hours of video into 1-2s chunks, uploading them to s3, then processing them in isolation?

Now that you place it like that, it does sound very dubious. The overhead required by uploading MBs go S3 go afterwards have to download it back to the lamdda's execution context should have the same order of magnitude as processing the payload. I'm sure they were aware of this but we're hoping that things would scale well due to the embarrassingly parallel nature of the problem.

that's how HLS works if you switch "processing" to "serving"