News for hackers. Stuff that matters.
Re:A bank runs serverless with PHP and AWS Lambda by willsmith72 on Wednesday November 01, 2023 @11:08AM (#38099246)
I always love a story about a successful strangler pattern migration.

I do wonder how they came to Lambda though. I love it for small workloads and highly variable demand services, but something like Treezor you'd think has relatively flat and high demand. The cloud cost for Lambda would be much higher than running the equivalent compute, even with something also highly scalable like ECS.

Re:A bank runs serverless with PHP and AWS Lambda by ceejayoz on Wednesday November 01, 2023 @11:16AM (#38099363)
We use Lambda (via Laravel Vapor; https://vapor.laravel.com/) for a big analytics batch processing job every night; it spawns hundreds of thousands of individual jobs that make various API calls, but only for a few minutes. For the rest of the day it's doing very little.

I'd imagine a bank has quite a few of these sorts of bursty bash processing needs; ACHes, end-of-day reconciliation, etc.

Re:A bank runs serverless with PHP and AWS Lambda by jackconsidine on Wednesday November 01, 2023 @12:07PM (#38100291)
What's your experience with Vapor? We played around with it and thought about switching from Forge, but it was too complicated for our codebase. I love the idea though.

How expensive our your cloud costs? CI/CD pretty good?

Re:A bank runs serverless with PHP and AWS Lambda by ceejayoz on Wednesday November 01, 2023 @12:12PM (#38100384)
Overall quite good. We use their Docker container support (to get ffmpeg, mostly) and it's been pretty smooth running throughout. I'd previously used a lot of Heroku so the habits developed there paid off in our codebase for Vapor; only a few tweaks were really needed.

Cost-wise we're pretty bursty, so it's saved us quite a bit. Lambda runtime is money, so a little bit of performance optimization can go a long way. CI we run through Github Actions; an environment secret plus `vapor deploy` and it's on its way up after a successful run.

Wasn't a fan of Forge; for that style of things I'd use Ploi nowadays.

Re:A bank runs serverless with PHP and AWS Lambda by ndriscoll on Wednesday November 01, 2023 @05:51PM (#38105823)
At my last job, we had an architecture that ran hundreds of thousands of individual jobs with internal API calls for nightly account updates and reporting work, and it was a major battle to finally get it replaced with some simple SQL scripts which ran orders of magnitude faster while being a lot more reliable.

As I understand it, banking commonly uses JVM where you can easily do async or multithreaded processing and task scheduling, so lambda doesn't offer much over ECS, and the timeout could be a real killer.

Re:A bank runs serverless with PHP and AWS Lambda by me_graf on Wednesday November 01, 2023 @06:53PM (#38106523)
I did some fintech work as a junior / kinda non junior dev. This is correct. I think 90% of the work was taking pictures of checks, and setting up batch jobs. all batch: - money movement (between accounts) - transfers - wires - etc.

even the 'non' batch things were done in batch (even if it's do now, and only 1).

Re:A bank runs serverless with PHP and AWS Lambda by theshrike79 on Thursday November 02, 2023 @04:34PM (#38119751)
Lambdas are amazing at clearing queues.

You can have a single queue gathering workloads and then have Lambda functions grab a bunch of tasks, handle them, grab more etc. until the max runtime is met and they're automatically rebooted.

It's also pretty trivial to add more processing capacity when the queue goes over a specified limit just by allowing for more concurrent Lambdas.

And when there's nothing to process, you've got one idling Lambda doing nothing and costing nothing.

Re:A bank runs serverless with PHP and AWS Lambda by cogman10 on Wednesday November 01, 2023 @11:43AM (#38099819)
This was my thought. Lambda works well for infrequent tasks. But hosting an API? That seems like it could get expensive pretty fast.
Re:A bank runs serverless with PHP and AWS Lambda by deleugpn on Wednesday November 01, 2023 @12:08PM (#38100301)
It's cheaper than most options where you need to pay a human to manage something with the same level of scalability and security that AWS provides.
Re:A bank runs serverless with PHP and AWS Lambda by ndriscoll on Wednesday November 01, 2023 @06:00PM (#38105925)
Lambda scalability for an API is awful; it can't run concurrent requests (of course PHP largely can't do this either). An ECS service with pretty much any other language (for async/multithreaded runtimes) will scale far better.
Re:A bank runs serverless with PHP and AWS Lambda by cebert on Wednesday November 01, 2023 @06:07PM (#38105999)
A single lambda can only handle one request at a time, but you can scale up 1,000s of instances if needed.
Re:A bank runs serverless with PHP and AWS Lambda by ndriscoll on Wednesday November 01, 2023 @06:19PM (#38106149)
A single minimal ECS instance can handle thousands of concurrent requests with an async runtime without running into so many cold starts that your p25 time is in the hundreds of ms.

1000 lambdas with 1000 database connections is also going to be a horrible way to do things on the backend.

Re:A bank runs serverless with PHP and AWS Lambda by WatchDog on Wednesday November 01, 2023 @06:53PM (#38106514)
ECS fargate is pretty easy to scale, and much more cost effective.

I personally don’t think running on EC2 is particularly difficult to manage, and it’s even cheaper again.

Re:A bank runs serverless with PHP and AWS Lambda by Sohcahtoa82 on Wednesday November 01, 2023 @12:54PM (#38101127)
Lambda is $0.0000166667 for every GB-second, which works out to ~$43.20 for 30 days. For a little more ($49/month), you could get a c6g.large and get two dedicated CPU cores (ie, not the shared burstable t2/t3) and 4 GB of RAM.

So yes, very expensive fast if you get much traffic at all. That c6g.large should easily be able to handle dozens (if not hundreds) of requests in parallel, especially if many of those requests are going to have to wait for results from a database.

If you have more than ~800 hours of Lambda execution time per month (30 days being 720 hours), then you're better off with the c6g.large. The only reason to choose the Lambda is if your traffic is extremely bursty and the c6g.large can't handle the load.

Of course, this is all assuming you're using a 1 GB Lambda. Likely, your Lambda needs less, in which case the amount of traffic needed to make the EC2 more worthwhile higher.

Re:A bank runs serverless with PHP and AWS Lambda by deleugpn on Wednesday November 01, 2023 @01:57PM (#38102175)
Then your EC2 instance suddenly stops working and your downtime cost you a fat contract that makes the cost of serverless a pocket change.

If you want to compare AWS Lambda with EC2, you need to bring in AWS Load Balancer, Auto Scaling, Security Upgrades for the OS, Healthcheck and many more.

Yes, Lambda is not needed in a lot of context, but if you have the expertise to use it and you want to provide enterprise-grade compliance, it's a walk in the park.

Re:A bank runs serverless with PHP and AWS Lambda by willsmith72 on Wednesday November 01, 2023 @02:14PM (#38102444)
Everything you listed is why I mentioned ECS though. To me it's a perfect in-between.

Bringing the long-lived server back also makes things like long-lived connections, memory caching, cold starts and database connections a non-issue again.

Re:A bank runs serverless with PHP and AWS Lambda by Sohcahtoa82 on Wednesday November 01, 2023 @02:19PM (#38102553)
Good points all around.

The costs certainly are not as cut-and-dry as I originally expressed it.

Re:A bank runs serverless with PHP and AWS Lambda by cebert on Wednesday November 01, 2023 @06:10PM (#38106033)
With Lambda you’re in three different Availability Zones by default, so you also need to factor running your proposed EC2 solution in 3 AZs too.
Re:A bank runs serverless with PHP and AWS Lambda by ndriscoll on Wednesday November 01, 2023 @06:09PM (#38106028)
2 cores and 4GB RAM should have no problem handling thousands or tens of thousands of concurrent requests on the JVM. Lambda's documented concurrency maximum is "tens of thousands"[0], which means an 8GB m6g.large or ECS instance can probably handle larger bursts than lambda can. If you had extreme bursts, you'd also be having almost every request hit a cold start.

[0] https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-...

Re:A bank runs serverless with PHP and AWS Lambda by pfych on Wednesday November 01, 2023 @08:44PM (#38107444)
I've been working with Lambda across a few large projects for their API's, with each endpoint being its own lambda.

This has been scaling really well for us and affordability wise it makes the development and testing phase basically free, with the final deployed costs almost never exceeding double digits monthly. This is for multi-az + multi-region deployments too.

Re:A bank runs serverless with PHP and AWS Lambda by mnapoli on Wednesday November 01, 2023 @12:13PM (#38100391)
Their clients have very variable traffic patterns because they are in very different industries. They have traffic spikes at lunch, or at the end of every month for example:

> infrastructure must be able to scale and be resilient to accommodate various usage patterns. Whether it's a luncheon voucher transaction spike at lunchtime or a monthly batch of transactions by corporate clients

Re:A bank runs serverless with PHP and AWS Lambda by tomrod on Wednesday November 01, 2023 @12:38PM (#38100865)
I'd be more worried about decimal data types being interpreted as floats. That could get gnarly fast on the wrong language runtime, and is partially why Cobol programmers are still in demand in the financial sector.
Re:A bank runs serverless with PHP and AWS Lambda by chx on Wednesday November 01, 2023 @05:19PM (#38105444)
the beauty of bref is you work with the same php engine as before. I am a (very) long time PHP developer, my primary work is still Drupal so the ability to make a small serverless app in PHP was very handy for me. My app implements a webhook and I feel Lambda is made for this: the script is small and quite fast, it checks the payload and puts an item into SQS as appropriate. Then another infra can slowly empty the SQS queue. The problem here is the load is insanely spikey -- sometimes tens of thousands of requests in a very short time frame, sometimes nothing. Directly writing this into a transactional database would require very costly infra. The Lambda-SQS model smooths it out. It's fine if some changes take minutes or even hours to be recorded.
Re:A bank runs serverless with PHP and AWS Lambda by Supermancho on Wednesday November 01, 2023 @06:45PM (#38106438)
> I do wonder how they came to Lambda though.

Probably some security certification that AWS Lambda comes with. I worked at Cambia Health that used Lambda (Javascript) for dealing with customer mobile data. It was a nightmare solution (cost and complexity) for what should have been a simple fan out queue and process system. They hired me to help fix it. Every other solution was a flat no. AWS Lambda was was HIPPA compliant (for whatever that's worth) and some other security assurances that I don't remember, so I was basically just another developer (with a moron for a manager).

Re:A bank runs serverless with PHP and AWS Lambda by iLoveOncall on Wednesday November 01, 2023 @07:13PM (#38106721)
> I do wonder how they came to Lambda though

Probably misplaced hype about being fully serverless.

As you said, ECS would be much more suitable while also addressing the issue that they had (spikes in traffic).

Lambdas behind API Gateway sounds like a good idea and even looks like one if you don't have experience with ECS and don't do a proper cost comparison analysis.

My team uses ECS' autoscaling to scale up to tens (or even hundreds) of thousands of tasks in minutes without issues.