Re:A shared file system for lambda functions
by jupp0r on Tuesday June 16, 2020 @07:17PM
(#23545523)
This is a horrible idea. This gives lambda functions shared mutable state to interfere with each other, with very brittle semantics compared to most databases (even terrible ones).
[ Parent
| on HN ]
Re:A shared file system for lambda functions
by riquito on Tuesday June 16, 2020 @07:34PM
(#23545680)
"practicality beats purity"
Depending on the use case it can be a good option (some are listed in the article). If the dev doesn't get the disadvantages the rest of the architecture will hardly be correct anyway
Re:A shared file system for lambda functions
by scarface74 on Tuesday June 16, 2020 @07:28PM
(#23545613)
I need this specifically as a part of a state machine. Most of the steps involve a Lambda loading and unloading csv data between S3, Redshift, and Aurora where no local storage is needed. The last step where we had to download the files locally and compress multiple files together was done manually via a script because they were greater than 512Mb.
We were just about to put the script in Fargate (Serverless Docker) and run a ECS task as part of the state machine. Now we don’t have to.
Re:A shared file system for lambda functions
by hansef on Tuesday June 16, 2020 @07:37PM
(#23545705)
This - if you have to fetch data from or output data to outside of the AWS ecosystem, the 512mb /tmp limit pushes you into the additional (relative) complexity of having to run on Fargate pretty quickly. Just had to deal with this for a content ingest job involving pulling a couple GB of data from an FTP server, processing it and pushing it into an RDS database on an hourly basis. Would have been super simple if the file was on S3 already.
Re:A shared file system for lambda functions
by jupp0r on Wednesday June 17, 2020 @08:15PM
(#23558387)
Where do you need a distributed file system in this use case? Sounds like all you need is some local scratch space?
Re:A shared file system for lambda functions
by scarface74 on Thursday June 18, 2020 @06:48AM
(#23561832)
Lambda only gives you 512MB pf local “scratch space”. If it had to provision and de provision gigs of space at each invocation, it would probably cause longer start times and shut down times.
Re:A shared file system for lambda functions
by jupp0r on Wednesday June 17, 2020 @08:14PM
(#23558381)
Why do you need a distributed file system to solve that problem? It sounds like all you need is some scratch space?
Re:A shared file system for lambda functions
by KaiserPro on Wednesday June 17, 2020 @02:50AM
(#23548524)
It can, but...
As you can seek() on NFS, you can quickly scroll to the middle of a massive file. Unlike S3 where you have to download the whole file first.
Re:A shared file system for lambda functions
by hamandcheese on Wednesday June 17, 2020 @04:21AM
(#23549101)
S3 allows you to specify a byte range.
It certainly has some potential for shooting oneself in the foot in some scenarios, but still nice to have the option