News for hackers. Stuff that matters.
Re:Amazon S3 Object Lambda by blunte on Saturday March 20, 2021 @03:57AM (#26521522)
We do this all the time already when we write various interface services to access a common data store. Each service provides whatever interface is appropriate and performs any desired transformations.

It's not clear to me what the real benefits of Object Lambda provides. Indeed it could move your transformation code closer to its data, but the cost of doing that is the management and complexity of the lambdas. I am curious what a best use case example would be for this.

Re:Amazon S3 Object Lambda by chrisandchris on Saturday March 20, 2021 @04:02AM (#26521540)
For my use-cases, this would be a huge benefit:

> Resizing and watermarking images on the fly using caller-specific details, such as the user who requested the object.

Now we need to either do rescale directly after storing or need to use another service (on EC2) to do the resize. With Object Lamda, we could do it on-the-fly as the user calls the image. This is operational a lot easier.

Re:Amazon S3 Object Lambda by ec109685 on Saturday March 20, 2021 @04:39AM (#26521679)
Even without this feature, you could stand up a lambda and do the resize on the fly.

Advantage here is that clients can just continue to use their existing S3 interface yet your lambda is still invoked.

Re:Amazon S3 Object Lambda by captn3m0 on Saturday March 20, 2021 @05:47AM (#26521892)
Does this get triggered over the S3-over-HTTPS (the one used for websites) served objects? (instead of S3 APIs)
Re:Amazon S3 Object Lambda by joshuanapoli on Saturday March 20, 2021 @09:41AM (#26523183)
The benefit of S3 Object Lambda is that your function is adapted to the S3 API. If you already have an application integrated with S3, then you can use Object Lambda to provide some new behavior to the data without changing the application.
Re:Amazon S3 Object Lambda by matt-attack on Saturday March 20, 2021 @08:33AM (#26522744)
How are parameters passed? That is how does Lambda get its resize parameters from just an S3 GET? Are they query params?
Re:Amazon S3 Object Lambda by jon-wood on Saturday March 20, 2021 @09:45AM (#26523197)
In the example use case for that they suggested baking the size into the filename, like image_500_240.jpg
Re:Amazon S3 Object Lambda by skynet-9000 on Saturday March 20, 2021 @12:51PM (#26524336)
You might want to build some rate limiting into your lambda as well!
Re:Amazon S3 Object Lambda by ralusek on Saturday March 20, 2021 @09:23AM (#26523048)
At the moment this is currently handled with lambda @ edge with Cloudfront. The benefit with Cloudfront over S3 lambda is that the image is automatically cached after the resize is computed.
Re:Amazon S3 Object Lambda by mwarkentin on Saturday March 20, 2021 @11:38AM (#26523866)
You could cache the image with s3 lambda too, no?

Edit: I guess you meant that cloudfront caches it for you automatically.

Re:Amazon S3 Object Lambda by ralusek on Saturday March 20, 2021 @12:55PM (#26524369)
Exactly, yup. Whatever it returns is automatically cached at the key.
Re:Amazon S3 Object Lambda by barbazoo on Saturday March 20, 2021 @11:48AM (#26523939)
Your example is way better than what AWS has on their page (marketing, analytics) which made me think immediately: you shouldn't use a file store for that but a database.
Re:Amazon S3 Object Lambda by lambda_obrien on Saturday March 20, 2021 @01:39PM (#26524669)
S3 is basically a database. You can store everything there and interact with it via SQL with Athena, for example. I could see utilizing all of Amazon's "infinitely scalable" services to build very simple and powerful software really easily.
Re:Amazon S3 Object Lambda by barbazoo on Saturday March 20, 2021 @01:47PM (#26524711)
I think Athena has to get a lot less expensive for that to become true.
Re:Amazon S3 Object Lambda by kryptk on Saturday March 20, 2021 @03:42PM (#26525588)
For small numbers of large queries, Athena has been an incredible technology stack .. I pay pennies a month to run my analytics jobs on it (aggregates across a few hundred million rows, joins with tens of millions of rows into hundreds of thousands of rows).

I start my data as .csv.gz but the first step is a CTAS to extract columns and convert to compressed parquet. This step basically costs the most but gives a 10x data size reduction to downstream steps.

Athena does not work at all if you perform large numbers of small indexed read queries, definitely use a traditional database for that.

Re:Amazon S3 Object Lambda by lambda_obrien on Saturday March 20, 2021 @07:57PM (#26527004)
Yes, it's great if you setup the data structures properly, we pay a few bucks a month to run a lot of infra on it to query telemetry data over a few hours to a few days at a time. I think an equivalent cost in just a database server instance to run the same load would be at least 25 to 50 a month. You're not really locked into athena, either, it just takes standard sql, so you could easily transition to a dedicated box later.
Re:Amazon S3 Object Lambda by chrisandchris on Saturday March 20, 2021 @03:57PM (#26525686)
I copied an example from the AWS blog post and adapted to our use-case.

I did never work with it, but there is Athena that allows to query S3 in something like SQL. I think there _might_ be some cases were S3 is a useful database, as one can use pre-signed URLs to let any client write to the database (S3). That‘s something I‘m not sure how to set up with a classic database and without any API in between.

IMHO, I think it‘s all about the same functionality with a simpler architecture.

Re:Amazon S3 Object Lambda by kbumsik on Saturday March 20, 2021 @07:43AM (#26522447)
Our team write complex code for Spark to sanitize/anonymize personal data before processing.

With this we might simplify the pipeline because we can keep the same S3 interface.

Re:Amazon S3 Object Lambda by mxz3000 on Sunday March 21, 2021 @09:08AM (#26530100)
This. This allows sensitive data to never leave the storage layer.
Re:Amazon S3 Object Lambda by SoapSeller on Saturday March 20, 2021 @05:43AM (#26521879)
If S3 is database - Object Lambda is stored procedure?

I guess that for some systems that are heavily S3-based it will help adding new features without needing to rewrite the interface or duplicate the functionality.

It's nice to know that AWS achieved such lock-in with some customers.

Re:Amazon S3 Object Lambda by lettergram on Saturday March 20, 2021 @10:02AM (#26523300)
> the cost of doing that is the management and complexity of the lambdas

The lock-in mechanism. Imo this isn’t more functional than other approaches, it’s a way to ensure you don’t switch services.

Re:Amazon S3 Object Lambda by mr_toad on Saturday March 20, 2021 @10:31PM (#26527683)
> It's not clear to me what the real benefits of Object Lambda provides.

It’s basically doing the same thing as an SQL view. It just hides and abstracts away some complexity.