Honestly, it reminds me of PHP development years ago: running it locally sucked, so you need to upload it to the server and test your work. It. Sucked.
With SAM we're talking ~6-7 seconds with an SSD to build + LOCALLY invoke a new copy of a fairly simple function where you're changing 1 line of code, and you need to do this every time you change your code.
That's even with creating a custom Makefile command to roll up the build + invoke into 1 human action. The wait time is purely waiting for SAM to do what it needs to do.
With a more traditional non-Serverless set up, with or without Docker (using volumes) the turn around time is effectively instant. You can make your code change and by the time you reload your browser it's all good to go. This is speaking from a Python, Ruby, Elixir and Node POV.
(I'm still doing the "upload to server to test" thing.... I've tried MAMP and Vagrant/VirtualBox for local dev but both of them seem horribly complex compared to what we can do with local dev with node.js/mongo and so on.)
Of course this won't fix the fact that you have a lambda behind api gateway that does some heic->jpg conversion and can't be hit outside the DMZ, or some esoteric SQS queue that you can't mimmic locally - but it should get you almost there.
sudo apt install nginx php mysql, point the www directory of nginx to something on your computer (/mnt/c/projects/xyz) and you've got a running setup. Or run Linux in general, that's what most people I've seen work on backends seem to do. You can run the same flavour of software that your deployment server runs so it'll save you time testing and comparing API changes or version incompatibilities as well.
I don't know any solution for macOS but you can probably get the necessary software via Brew if you're so inclined. Then run the built-in PHP web server (php -s) and you get the same effect.
Apache is almost as easy, download zip, unpack & configure apache conf to use your php.
MySQL is somewhat more complicated because you need to run an setup script after unpacking the zip.
Disclaimer: I work at AWS, however not for any service or marketing team. Opinions are my own.
This is what I tried to do initially after experiencing the dev pain for only a few minutes.
But unfortunately this doesn't work very well in anything but the most trivial case because as soon as your lambda has a 3rd party package dependency you need to install that dependency somehow.
For example, let's say you have a Python lambda that does some stuff, writes the result to postgres and then sends a webhook out using the requests library.
That means your code needs access to a postgres database library and the requests library to send a webhook response.
Suddenly you need to pollute your dev environment with these dependencies to even run the thing outside of lambda and every dev needs to follow a 100 step README file to get these dependencies installed and now we're back to pre-Docker days.
Or you spin up your own Docker container with a volume mount and manage all of the complexity on your own. It seems criminal to create your own Dockerfile just to develop the business logic of a lambda where you only use that Dockerfile for development.
Then there's the whole problem of running your split out business logic without it being triggered from a lambda. Do you just write boiler plate scripts that read the same JSON files, and set up command line parsing code in the same way as sam local invoke does to pass in params?
Then there's also the problem of wanting one of your non-Serverless services to invoke a lambda in development so you can actually test what happens when you call it in your main web app but instead of calling sam local invoke, you really want that service's code to be more like how it would run in production where it's triggered by an SNS publish message. Now you need to somehow figure out how to mock out SNS in development.
Serverless is madness from start to finish.
SAM already provides a way to mock out what SNS would send to your function so that the function can use the same code path in both cases. Basically mocking the signature. This is good to make sure your function is running the same code in both dev and prod and lets you trigger a function in development without needing SNS.
But the problem is locally invoking the function with the SAM CLI tool is the trigger mechanism where you pass in that mocked out SNS event, but in reality that only works for running that function in complete isolation in development.
In practice, what you'd really likely want to do is call it from another local service so you can test how your web app works (the thing really calling your lambda at the end of the day). This involves calling SNS publish in your service's code base to trigger the lambda. That means really setting up an SNS topic and deploying your lambda to AWS or calling some API compatible mock of SNS because if you execute a different code path then you have no means to test the most important part of your code in dev.
> In the case of Postgres, you could use an ORM that supports SQLite for dependency-free development but at a compatibility cos
The DB is mostly easy. You can throw it into a docker-compose.yml file and use the same version as you run on RDS with like 5 lines of yaml and little system requirements. Then use the same code in both dev and prod while changing the connection string with an environment variable.
> That’s not exactly a knock against serverless itself, is it?
It is for everything surrounding how lambdas are triggered and run. But yes, you'd run into the DB, S3, etc. issues with any tech choice.
And yes, I think the problem here are APIs you use in multiple places but can’t easily run yourself in a production-friendly way. Until AWS builds and supports Docker containers to run their APIs locally, I don’t see how this improves... end to end testing of AWS requires AWS? ;-)
Of interest, I've spent some free time crunching on CNCF survey data over the past few months. Some of the strongest correlations are between particular serverless offerings and particular delivery offerings. If you use Azure Functions then I know you are more likely to use Azure Devops than anything else. Same for Lambda + CodePipeline and Google Cloud Functions + Cloud Build.
It’s hard to do this when surrounding code doesn’t accommodate the approach, but it’s great way to design an application if you have the choice. I really love sandboxing features before moving them into the application. Everything from design to testing can be so much faster and fun without the distractions of the build system and the rest of the application.
What I have now is a loosely coupled, serverless, frontend+backend monorepo that wraps AWS SAM and CloudFormation. At the end of the day it is just a handful of scripts and some foundational conventions.
I just (this morning!) started to put together notes and docs for myself on how I can generalize and open source this to make it available for others.
stack is vue/python/s3/lambda/dynamodb/stripe but the tooling I developed is generic enough to directly support any lambda runtime for any sub-namespace of your project so it would also support a react/rails application just as well.
I don’t think it’s as bad as people make it out to be.
Sure, you can write a bunch of tools to work around the crufty, terrible development environment's shortcomings. But ultimately, you are just locking yourself further & further & further in to the hostile, hard to work with environment, bending yourself into the bizarre abnormal geometry the serverless environment has demanded of you.
To me, as a developer who values being able to understand & comprehend & try, I would prefer staying far far far away from any serverless environment that is vendor locked. I would be willing & interested to try serverless environments that give me, the developer, the traditional great & vast powers of running as root locally that I expect. Short of a local dev environment, one meets both vendor lock in, & faces ongoing difficulties trying to understand what is happening, with what performance profiles/costs. I'd rather not invest my creativity & effort in trying to eek more & more signals out of the vendor's black box. Especially if trouble is knocking, then I would very much like to be able to fall back on the amazing toolkits I know & love.
That isn't a compliment.
Time will tell.
but lambda gets to the point where there is no local parity, where it's detached, is no longer an easier managed (remotely operated) parallel to what we know & do, but is a system entirely into itself, playing by different rules. one must trust the cloud-native experience it brings entirely, versus the historical past where the cloud offered native local parallels.
Personally for simple projects I've had pretty good experiences writing a Yaml-based template directly, and for more complex projects I use Troposphere to generate Cloudformation template Yaml in Python.
If you are working on a project where all of your infrastructure will live on AWS I would definitely urge you to give it a second look. The amount of infrastructure I manage right now with a single .yaml file is really killer.
Did you chose python for backend dev? Any framework like flask or django?
Also no preference of single-language for both backend and front-end, by using node.js backend?
Just trying to get into web-dev.
If I could do it all over, I would still choose Python. That being said, I have been working professionally (building apps like this) for almost 14 years so my willingness to bite off a homebrew Python framework endeavor as I did here is a lot different than someone just getting into the field.
Django: avoid unless you have a highly compelling (read: $$$$) reason to learn and use this tool. I cannot think of one, honestly.
Flask: fantastic, but be conscientious about your project structure early on and try to keep businees-logic out of your handler functions (the ones that you decorate with @app...)
Sophisticated or more sugary Node.js backends are not something I have ever explored, aside from the tried-n-true express.js. I tend to leverage Python for all of my backend tasks because I haven't found a compelling reason not to.
Minimalist frameworks are great for either very small (since they don’t need much of anything) or very large projects (since they will need a bunch of customization regardless).
In that regard, I think Django is kind of like the Wordpress of Python.
All-in-all, Django is not bad software. I have a bad taste in my mouth though because as I learned and developed new approaches to solving problems in my career I feel like Django got in the way of that.
For instance, there are some really killer ways you can model certain problems in a database by using things like single table inheritance or polymorphism. These are sorta possible in Django's ORM, but you are usually going against the grain and bending it to do things it wasn't really supposed to. Some might look at me and go: ok dude well don't do that! But there are plenty of times where it makes sense to deviate from convention.
That is just one example, but I feel like I hit those road blocks all the time with Django. The benefit of Django is it is pre-assembled and you can basically hit the ground running immediately. The alternative is to use a microframework like Flask which is very lightweight and requires you to make conscious choices about integrating your data layer and other components.
For some this is a real burden - because you are overwhelmed by choice as far as how you lay out your codebase as well as the specific libraries and tools you use.
After your 20th API or website backend you will start to have some strong preferences about how you want to build things and that is why I tend to go for the compose-tiny-pieces approach versus the ready-to-run Django appraoch.
It's really a trade-off. If you are content with the Django ORM and everything else that is presented, it is not so bad. If you know better, you know better. Only time and experience will get you there.
You have to roll your own way too often in Flask et al, so much so that I don't see any reason to use Flask for anything other than ad-hoc servers with only a few endpoints.
There's a couple of sharp edges still but in general it just 'makes sense'. If you don't like TypeScript there are also bindings for Python and Java, among others, although TypeScript is really the preferred language.
Currently using some CDK in a production app and finally I found a way of doing IaC I actually enjoy.
CDK is moving quite fast and not all parts are out of the experimental phase, so there are breaking changes shipped often. I think in a couple of years it will stabilize and mature and become a very productive way of working with infrastructure.
It's a little incomplete, missing some of the AWS IAM automation that makes local development smooth, environment management for testing and promoting changes, and some sort of visualization to make architecture easier to design as a team.
I work for a platform company called Stackery which aims to provide an end-to-end workflow and DX for serverless & CloudFormation. Thanks for comments like these that help identify pain points that need attention.
I tried AWS, and then IBM's offering which is based on an open source (Apache OpenWhisk) project, thinking that it might be easier to work with, but that was also a pain.
I just lost interest as I was only checking it out. For something constantly marketed on the ease of not having to manage servers, it fell a long way short of "easy".
Look into Firebase functions. Drop some JS in a folder, export them from an index.js file and you have yourself some endpoints.
> exports.helloWorld = functions.https.onRequest((request, response) => { response.send("Hello from Firebase!"); });
The amount of work AWS has put in front of Lambdas confuses me. Firebase does it right. You can go from "never having written a REST endpoint" to "real code" in less than 20 minutes. New endpoints can be created as fast as you can export functions from an index.js file.
Being able to throw up a new REST endpoint in under 10 minutes with 0 config is really cool though.
And Firebase Functions are priced to work as daily drivers, they can front an entire application and not cost an insane amount of $, per single ms pricing. Lambda's are a lot more complicated.
I've been pretty happy with Cloudflare Workers.
You can easily define environments with variables via a toml file. The DX is great and iteration speed is very fast. When using `wrangler dev` your new version is ready in a second or two after saving.
TJ abstracts the Lambda nonsense so you just get to use a standard HTTP res/resp interface. https://github.com/apex/up-examples/
Deployment iterations for me in Singapore can be as low as 2s, but tbh I use https://github.com/codegangsta/gin locally for development iterations.
I tried AWS Lambdas a few years back and it felt way more primitive.
My only nitpick, and only specifically relating to dotnet, is that config files and env vars differ between Functions and regular ASP.NET Core web apps. I think there is some work going on to fix that, but it's taking forever.
Edit: CloudFormation was also painful for me, the docs were sparse and there were very few examples that helped me out.
Over 5000 pages of documentation
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...
> very few examples that helped me out
AWS provide an example for most common services, plus there are thousands of other community supplied examples out there.
Yes there are examples, but there wasn’t one at the time that mapped to what I was trying to accomplish. Because, again, SAM templates are not one-for-one CloudFormation templates.
I found the community around SAM to be very limited. One of the many reasons I’ve moved to the Kubernetes ecosystem.
Still very actively working on this but our goal is that nobody ever has to test via deploy again.
Slightly OT... perhaps cheeky... any idea why firebase doesn’t provide automated backups for firestore and storage out of the box? Seems like a no brainer and a valuable service people would pay for.
DISCLAIMER: I used to be a developer of this product (it's from the previous company I've worked)
I was planning on trying out FastAPI + Zappa, but have not gotten round to it yet.
> serverless invoke local
All the time for doing testing. You can pass in a json file that contains a simulated event to test different scenarios. Works very well for us.
I believe they decided to re-create their own credentials chain and a lot of things are not working with MFA (like support for credential_process).
Firebase is magic... but I never recommend it for anyone, until there's some sort of migration path.
With Cloud Firestore (our serverless DB) that’s the case as well. And Firebase Auth can be seamlessly upgraded to Google Cloud Identity Platform with a click.
However you’re right that for many Firebase products (Real-time Database, Hosting) there’s no relation to Cloud Resources.
To be fair, Firebase recently released a local development tool which alleviates the need to deploy on every change, but I haven't used it yet.
Super useful though!
What does it mention?
Have you tried Amplify?
Unfortunately, I quickly hit the limits of its configurability (particularly with Cloudfront) and had to move off it within a few months.
Yes, SAM sucks. Boto3 sucks. I mean, yes, the AWS APIs are not the most obvious or ergonomics thing.
At least on AWS, the "SAM" experience has been probably the worst development experience I've ever had in ~20 years of web development.
It's so slow (iteration speed) and you need to jump through a billion hoops of complexity all over the place. Even dealing with something as simple as loading environment variables for both local and "real" function invokes required way too much effort.
Note: I'm not working with this tech by choice. It's for a bit of client work. I think their use case for Serverless makes sense (calling something very infrequently that glues together a few AWS resources).
Is the experience better on other platforms?