Sounds like its working for you, which is great, I guess.

However in general what you're doing isn't devops, its 'devs creating a big old mess for someone else to cleanup later'.

All you're doing is trading long term maintainability and quality for 'getting it done right now'.

Sure; it works. I get it. ...but I've also been on the receiving end when that 2-person team has 1 person leave and gets scaled up to a 5-person team, and you have to a) bin everything, and b) write the entire thing from scratch.

I personally consider it a very selfish way to build things. Me first. Me now. ...someone else's problem later.

There are scalable solutions in this space (eg. pipelines, airflow, azure data factory, glue); doing it using lambdas is... not. ideal.

That's not a problem with severless in general; but what you've described isn't any of the things which is good about serverless either: what you've described is the problem with the low barrier to entry with serverless, resulting in production serverless code that skips past 'quality control' to 'let's deploy straight to production!'.

As someone in charge of cleaning up the mess in a company that has 800+ employees and 300+ developers, I can only agree.

My company's infrastructure was never tended to and was apparently a free for all for a few years. A year or two ago, it was finally decided the free for all needed to end and all the developers saw their SSH root access removed.

My team of 5 SREs is now facing a constant backlash from developers frustrated at the rotten infrastructure, constant bugs, and impossibility to ssh into machines. We're slowly fixing the mess, migrating to Kubernetes and things are starting to get better, but god, how painful it is.

So yea, in its infancy a startup's infrastructure can't be perfect, but every single shortcut you're taking now, you'll pay double down the line. Choose your path wisely.

This assumes that survival is guaranteed if you do it the right way. Long term thinking can work in firms that are larger. For smaller firms with limited resources, motivation and technical expertise getting velocity is crucial. This is one of the reasons startups run into issues when they have to scale. I think it has to be a well thought out balance of speed and quality that is required in such environments.

That's such a straw man, it pains me to even respond to it.

Doing it 'quick and dirty right now with the tools I happen to know' instead of 'doing some research, then doing it in a way that is maintainable using standard tools and taking a bit longer' isn't 'endangering the survival of the company', it's professional level practice.

Failing to do so is, a) negligent, and b) incompetent.

I'm certainly not suggesting the OP isn't competent; they seem on the ball, and maybe serverless is the best choice for them because (insert reasons here)...but I think its fair to say, if you're using your one tool (ie. serverless) for all your tasks, you probably haven't done your due diligence.

Specifically in this case, using a standard ETL tool to do ETL type tasks... would probably be suitable.

Anyhow; no. Long term thinking isn't what this is about; it's about behaving like a professional when you do a professional job.

You pick the label you want to wear as an engineer, because, your practices define you.

You are on such a high horse, it pains me to even respond to you.

You can call it devops, a mess, whatever you want... but nothing about serverless frameworks is creating that mess. A service is a service, whether it's running Lambda, Spring Boot, or whatever the hottest new framework is... Nothing prevents you from documenting a service. Nothing prevents you from creating basic alarms (it's literally easier for me to create these alarms because of CDK than it is to alarm on the standard service framework I use).

We get it, you don't like serverless. But you're going to rag on OP and question their competence, and then drop some "you pick the label you want to wear"? Come on dude, seriously?

You're strawmanning too here... Can you show me where OP said they are using one tool for all of their tasks? they literally said "for one-off, little processes."

I cant imagine what it's like to be this arrogant.

If you read my comment, you’ll realise that I’m not saying serveless is bad.

I’m saying; if that’s your only tool, you’re not doing your job.

“Velocity” is not an excuse for not doing your job to a professional degree. Its a strawman people use to justify doing substandard work (or, often, pressuring engineers who know better, into doing substandard work).

While I think that there is definitely a worry of it turning into a big old mess (especially if they start depending on and talking to each other), I don't really know that I'd consider cron-job style independent one-off tasks any more maintainable in a monolith. I also don't know that I'd consider that specific to serverless (it's easy to create a ball of mud in anything).

All of our serverless projects are located in a single repository, and each little service gets its own little google doc that explains what it does, what functions it has, and any resources that it uses.

As mentioned below, as soon as an ETL gets multiple steps I agree that pure-lambda is not the right tool for the job (Airflow would be my preference, though we've also successfully used Step Functions)

How is owning an AWS account easier than a cheap dedicates server? You should have a code repo and backups in either case.

At least some people mention "but I can scale this lambda x1000" and that's one advantage... But you can do all those tasks on a hetzner server in same amount of type, just python scripts.

Because that "cheap dedicated server" isn't all that cheap. Just the documentation for backups on your Hetzner server is ridiculous. Why, in the 21st century, should I as an application developer ever have to worry about that?

Let's say that Hetzner server goes completely toast at 2am and you get a pingdom message. You log in, realize you have to restore from backup. You fire up a new instance, and go through your restore procedure, and everything works great. You're lucky to be offline for an hour.

I cannot even tell you what would happen if an underlying instance running a Lambda goes offline, because it never happens. But let's say that an underlying machine dies in us-east-1. Lambda just starts running more instances of your code on another underlying group of cpus, and you aren't offline.

And I’d argue the mere possibility of the server breaking at any time contributes to non-negligible amounts of sustained psychological stress.

Yes, this exactly. The fact that I can guarantee that my code is running, and have entire massive teams at Amazon whose literal job title is to keep my code running, it is such a freeing feeling. I might be able to delete rows from Dynamo from my code messing up, but Dynamo will never go offline.

The psychological effects of technology seem often overlooked—and they’re hard to quantify.

I wish there was some easier way to quantify the “soft” aspects of everything that surrounds us, e.g. the long-term impact of beautiful and usable UI design, the reduction in “ambient” psychological stress, the impacts of chaos/consistency on how we feel, any robust measures of happiness, excitement, relaxation, etc.

They seem to be such important attributes, but not as easily measured as a latency. Which in turn makes reaching consensus harder—and probably why design by committee doesn't work?

My joke around the office is some people optimize for latency, some for throughput, some for memory usage. I, however, optimize for sleep.

My entire goal is for my services to never wake me up. And if they do it had better because I clearly messed up and am the sole human alive able to fix it.

I optimize for escape hatches. Because if everything is on fire I want the ability to be able to bail(fix it).

There's no point in waking up when everything is blowing up if you don't have the access to fix the underlying issue. AWS doesn't actually guarantee 99.9% uptime. What they guarantee is they'll give you some company currency[1] if they can't meet or exceed 99.9% uptime.

So the next time the ship is on fire, don't worry, stay asleep and your Amazon account manager will be by shortly with a thimble of water to throw on you :)

[1] - https://aws.amazon.com/about-aws/whats-new/2019/03/aws-syste...

> I, however, optimize for sleep.

Bravo.

Having been on call for some truly monstrous amount of things and seen things break in all kinds of ways... I can't agree more with this principle.

While the cloud isn't cheap, if your org can afford it, there is no question about it: go with managed services. Focus on building things, leave the OPS to AWS/GCP/Azure.

We optimize for good engineers

> Dynamo will never go offline.

That's some ridiculous propaganda. AWS has whole regions going down for hours.

Are you really sure?

Tracking exactly what individual db entries ran and didn't run when there's one of these "serverless" "painless" interruptions to an ETL, ensures that you have an endless number of ever-changing status columns and re-run processes.

Are you really sure that it all re-ran? No. You have to query your latest batch - which means you have to have arbitrary boundaries for when failures might have occurred like say, last 24 hours. If was a weekend, just query the whole database looking for any entry that's missing the "final" status success. Try not to run anything while you're doing this, unless your system is tolerant to the DB being locked for a full-table scan.

This is all very fragile and change-averse. Who would choose to run an ETL like this? I have worked for large companies who all end up with this rickety system. Every one, every time. God forbid you want to run tests with a mock set of serverless AWS services.

Here's the beautiful thing about thinking serverless. "Are you really sure that it all re-ran?" Yes I am, because Dynamo streams or Kinesis streams guarantee message delivery. If my Lambda code has a bug in it, which yes sure that happens, I can as part of my exception catching, put that message into a dead letter queue and process it later or report on it. No polling, no querying, I know exactly what went wrong and when.

Meanwhile, I don't have to do things like /make sure the ETL process is running/ or /clear out a bunch of logs when a disk inadvertently fills up/ or any of those other stupid last-century compute problems. It's just a far, far easier way of thinking about application development.

> Dynamo streams or Kinesis streams

I can appreciate that. A lot more people than you might think, use Kafka or SQS because of various requirements (like that's all their infrastructure supports) and you end up with a massive "source" DB which is a duplication of all-time messages. Such is the reality of a larger byzantine organization. If it can't be audited by someone else, at their own pace, it's not approved.

I generally agree that if you're setting up multi-step or "enterprise grade" ETL processes that it's a job better handled by something like Airflow (we've done a few smaller ones with Step Functions which hasn't been terrible, but Airflow is still better).

Our use of AWS Lambda for ETL is usually just one-off small processes that don't have dependencies.

As soon as you start getting into "This ETL depends on X, Y, and Z being run first" I think you're out of "small utility" territory.

This is so naive. Unless you’re just creating utilities all day long, at some point you’ll actually have a production service that people are using that needs to be monitored, even if it is build out of lambda functions. And unless you’ve run AWS vs colo at scale, you have no concept of the orders of magnitude cost savings on bandwidth (at 95th percentile billing vs per gig) and compute.

I have run AWS vs colo at scale. For the cost of precisely 1 engineer fully loaded, I can have 500TB of bandwidth per month, every month. Guess what, it isn't going to take just 1 engineer to maintain your homegrown worldwide CDN.

Does it make sense for FAANG to run their own datacenters? Yes of course. Does it make sense for you? No. This is what I don't get about arguments for running a bunch of VPSs-- for literally any environment that isn't thousands of engineers, it is more cost effective to let someone manage your underlying infrastructure for you, and any workload at that scale, you aren't contracting with some random bare metal hosting shop.

You seem to forget there is are a ton of people who grew up on bare metal and know it quite well. And they can squeeze far more performance out of it then you could ever hope to see with outsourcing business-critical infra to a supplier who literally doesn't give a shit about you.

Have fun with an army of one-trick pony hires. One old-school diehard computer nerd knows enough to supplant 10 of those fools, but only costs 2x.

And many of them already work for AWS, who I can pay a small premium to to take advantage without having to hire one myself and then worry about replacing them when they decide to leave.

I think the point is sometimes that AWS small premium is really large. It varies a lot. For some the AWS costs are free or tiny. For others, AWS costs are larger than salaries, and it creeps up over time.

About staff, maybe consider finding a good consulting/support firm that can look after your servers for <10% the price of hiring one person, are friendly and on-call, and will likely stick around even as their people change.

That solves the worry about replacing people, as well as the cost.

But as with people, it can take some luck to find a good one :-)

(I used to provide that kind of consulting/support service. Not as much now because there isn't much demand, and development & research work is more satisfying. But still a little, ticking along in the background.)

Maybe it's hubris, but I'm 99% sure I could run a homegrown worldwide CDN for the cost of one engineer salary. I'd be willing to take that bet from anyone who thinks it can't be done.

You're forgetting that a huge part of that is maintaining accounts at a bunch of different colos and maintaining relationships with account managers and shipping hardware and all of the non-engineering work involved in running a CDN.

You'd be spending more than 1/2 your time just on the phone talking to people.

I definitely have not forgotten this aspect of it. Let's just say I've spent a lot of time on the phone for fun

If we want to talk about running a CDN company in terms of lots of additional organizational complexity in terms of customer service it can't be done on a single engineer's salary. My perception of a homegrown CDN is one that rivals uptime and deliverability metrics of existing commercial CDN providers. Within an existing org it can be done. If we're talking outside of an existing org think http://www.nosupportlinuxhosting.com/ to shave costs.

If we're willing to define parameters on what level of service this CDN needs to meet and ramp up time I'd be willing to take it on.

If you are not in the CDN business, why on earth would you want to do this?

This is my point-- I am not in the hosting business. I am in the application creation business. Every minute that I even have to think about modifying /etc/localtime in order to make sure that the instance is running in UTC is a minute I'm not being productive. Multiply that by those million little things that you have to do in order to keep even an ec2 instance running, and it's such a colossal waste of time and effort. Similarly, I am not in the database tuning business. I plan out my data relationships, create the required Dynamo table or tables, and I never need to worry about replication or backups or any of those ancillary tasks that keep me from being productive.

This is a slightly different conversation than the original but I'll give it a shot.

I'm not a woodworker, machinist, farmer, or doing a PhD in literature. Nevertheless, I engage in all of these things. I enjoy crafting things and learning new techniques. I enjoy learning how to optimize plant growth. I enjoy reading old literature and gathering what I can from it about the historical context, linguistics, etc. All of these types of things, while not "productive" give me additional knowledge that I can pull from in "unrelated" tasks.

In the tech world, I'd say that realistically all abstractions are leaky. So engaging in these tasks(you mentioned) are productive. But even more than that because abstractions are leaky this is knowledge that is useful. If you don't have the knowledge of the systems underlying the abstractions you're using it's a footgun waiting to happen.

I can give an example. I had a boss that was doing some serverless work and he made an assumption about JSON. I told him in review that JSON does not respect ordering of keys. Well, he assumed it would not be a problem. It was pushed to production anyways. And a few weeks later it came back to bite him.

Now what happens when you don't dig into your abstractions? You lack the knowledge to fix problems that happen at a lower level. When you're deploying these "relationships" to your database provider. What happens when the machinery breaks down for replication, backups, etc? You have to sit around and wait for an expert to fix your app. Meanwhile you're staring at your users and stonewalling.

No, you're not in the hosting business, you're not in the database tuning business, you're not in the sysadmin business, and you're not in the Ecmascript working group. Until you are. Then when you're foisted into this role you're a fish out of water. Specialization is for ants, but we're humans, not ants.

Yep, something like terraform + your favorite config management tool and monitoring system would do nicely for a mostly automated setup once done. I am guessing that a really nice setup could be created in under a month.

But are you including your own salary into that cost?

I believe he is comparing aws bills vs. engineer time+other hosting bills.

Depends on scale of course. There will always be some company that isn't of sufficient size to make it worth the cost.

This is how we actually end up having centralized services. The larger companies who have all this infrastructure, and have the size to justify building their own.. well they inevitably ask themselves "Hey! we have all this infra, we have to pay these people anyways, why aren't we selling excess capacity?"

Then you get AWS. Then as time progresses people at even large institutions that could benefit from running their own infra just go and use another company's infra. Where the other company has evolved into being an infra provider instead of a car parts store, book store(amazon), or search engine.

To answer your question directly. Yes I think this can be done within one engineer salary even if I'm including my own salary. There would be a startup cost that eats into that salary in year 1. But it would be manageable. And then iterative upgrades/replacement parts would be a negligible cost ongoing after year 1. With some financial tricks that startup cost can be spread out over a long enough period that it wouldn't even impact year 1.

Maybe you’re extraordinary, and for most people it’s not worth it.

I wouldn't say I am. But I think the perceived complexity(fear) of managing hardware is overblown. However as time goes on and people become more afraid of it due to the loss of institutional knowledge it won't be an overblown fear anymore. At that point institutions will have no choice but to contract out their infrastructure to a centralized institution or an oligopoly of institutions.

And we're approaching that because as these types of skills are outsourced to these institutions the amount of people with the skills necessary to carry them out are diminished in number from attrition. Amazon doesn't need 10 company's worth of engineers to have the requisite skills. If everyone buys into this model then eventually the market will only bear the number of engineers having that knowledge as Amazon needs.

extraordinary here would be either a developer with solid Linux admin skills, or a Linux admin with basic developer skills.

Nobody’s saying serverless makes monitoring unnecessary. But a serverless architecture gets rid of an entire category of bureaucracy and issues that can arise from dedicated servers.

What about the psychological stress of a massive AWS bill because of a misconfiguration by you or someone on your team?

A very good point. Exchange the risk of a problem taking you offline, losing sales, for the risk of a problem not taking you offline, costing you a fortune.

That's a big selling point for DigitalOcean: you sleep easy knowing what your bill will be. If you have an unexpected spike in traffic--whether a great opportunity, a mistaken test run wild, or a DDoS--it doesn't increase your bill. AWS offers the opposite: no matter what happens, we'll keep you online and just send you the bill.

A middle ground between both would be ideal. Does AWS offer settings to cap your spendings? (I googled and found AWS Budgets but it seems to be only an alerting system, not a cap.)

AFAIK at the moment there is no cap. It's pretty much like going long on a crashing market. However, you can later have a bunch of emails and tweets basically begging to be reimbursed and it works most of the time.

Going long will limit your losses to the amount invested. The point is that your AWS bill has no such limit whatsoever.

Yes, going with AWS is a short. You're shorting your own popularity. If your popularity explodes upward, there is no cap on your losses to AWS. The theory is that you are also hedged by your business so the limitless losses to AWS are more than compensated for by even larger gains in your business. But if you're caught with an unhedged short....yikes.

The recommended solution is, ironically for this topic, to have your billing alerts trigger Lambda functions you've written to shut down your infrastructure.

It’s acute.

My psychological stress is knowing our company is hemorrhaging 50k €/month on AWS fees (mainly Lambda, step functions, and CloudWatch) to run a production app that could easily run on a couple of VMs (for failover) and a managed DB instance.

Edit: add RDS to the parenthetical list of AWS services above.

> Just the documentation for backups on your Hetzner server is ridiculous.

Nothing a server template, code repo, and daily db dump cronjob can't solve

I’m assuming this is a joke, but this is exactly what most people who just want to run some code don’t want to deal with.

I like how people here pretend like this is some hard task that should only be done by experts when I know fifteen-year olds who figured this out all by themselfes for backing up their game servers

It's not monumentally hard or anything, but it still takes time and attention that I don't want to spend.

For a small business, it makes sense to outsource things like that, even you if you could conceivably do it in-house.

Then maybe they should learn how to deal with it

Learning how to deal with it is great, until you find out what you don't know.

Dealing with server infrastructure isn't business differentiating. Doing it poorly can certainly be business terminating, though.

> Dealing with server infrastructure isn't business differentiating

I agree, for most businesses no it isn't.

However, saving on cloud hosting costs to replace them with a handful of dedicated servers in different data centres can make a large difference to costs, depending on what the cloud usage is. For a funded rocketship startup with lots of free AWS credits it makes sense to use AWS; for a steady state or long-term low income business, not so much.

Those costs can make all the difference to a business if the cloud rental is creeping up to look comparable with salaries.

I've used a combination of dedicated and cloud hosting for a long time. In general the dedicated systems are so much cheaper for the same amount of bandwidth and compute, and you can run VMs on them with great performance, so you can run everything interesting inside VMs and a lot of the arguments around upgrade downtime have completely gone away.

With some cloud to assist, you can even use cloud temporarily to cover the downtime for major upgrades such as replacing hosts or reconfiguring the cluster network.

99.9% uptime with excellent bandwidth is easily achievable this way, and you can completely avoid user-facing downtime due to planned upgrades, with a bit of planning around DNS and IP transitions, VM migration, overlay networks, things like that.

Sometimes at much lower opex cost (100x) than the modern cloud-recommended equivalents, although it takes knowledge and time to ensure it.

If you have enough redundancy and occasional infrastructure stress testing, a similar level of "optimise for sleep" is possible as with cloud deployments. If you care about this (not every business does), it does take some work and knowledge to produce and verify hot redundancy, and of course there are costs.

Things will go wrong, but they'll tend to be at the level of things running inside VMs and containers, rather than hosts failing. The same things which go wrong with cloud deployments anyway.

The cloud can be used alongside dedicated to provide some of that redundancy and extra scaling. Cloud is rented in small units of time, so you can use cloud as "free most of the time" cold-backup service, and if there's a burst in traffic beyond what the dedicated systems can provide, or if you want to temporarily improve geographic locality for a service.

It might seem wasteful to rent larger dedicated services rather than cloud-as-needed, but the cost differential has changed in the last 10 years to favour the former more than it used to.

Are you taking into account the salaries needed to do this?

Even assuming that you save 100% of your hosting costs, you have to be spending quite a lot to even cover the salary of one full-time employee.

I can believe that larger businesses can save quite a bit, but I don't see how it makes sense for anyone spending less than seven figures a year on AWS.

This is a peak HN comment right here.

I used snapshots. It was super easy. They can do a snapshot every day and keep N copies. The cost is easy to factor, not a runaway bill.

>Just the documentation for backups on your Hetzner server is ridiculous. Why, in the 21st century, should I as an application developer ever have to worry about that?

I dunno, that's a problem for the ops team. It was in the 20th century too. The only reason you would have to worry about it is if you're pulling double duty as under-trained ops. If that were the case the right answer to every "build or buy" question is "buy" because you don't have anyone with the domain knowledge to bring more architectural options to the table than 'half-ass server management with internet tutorials.'

So get a cheap EC2 machine instead. There are reasonable steps between “buy your own hardware” and “everything is a Lambda”.

That's why you have three hetzner servers, so if one goes toast, you still have a redundant operation as you can easily failover while you recover the other host.

> Why, in the 21st century, should I as an application developer ever have to worry about that?

Then don't have backups at all, let's see how that goes.

Mostly it's about the maintenance costs of owning and managing a server -- on Lambda most of these tools are "set and forget".

I don't want to have to manage security updates, disk usage, or any of the other multiple things that come with owning a server.

I also could probably debate on the friendly-ness of just dumping all of these one-off services onto a single server (seems like it could get messy pretty quick), but I think you could probably go either way on that.

We are also an AWS shop for our "production" stack already, so there is no added overhead for owning an AWS account.

Servers can also be "set and forget", just enable automatic patching

Because you need two servers.

One to run the application.

One to run the application while the first is being upgraded.

Now you need three servers, one to run the application, one as backup and one as load balancer.

99.99% uptime means you can be down no more than an hour a year. Which is really easy to overshoot when you're dicking around with a dist upgrade on three linux boxes.

So that's why you end up with a server farm starting from your one small server.

Basically nothing I have ever used had 99.99% uptime except maybe the phone and emergency services. Why do you need this level of uptime unless life and limb depends on you?

My bank has regular scheduled downtime, Steam/Dota has regular downtime, entire Microsoft datacenters had issues and had around a day of downtime, Lime, Uber, my mobile carrier - all had downtime or bugs or glitches that were equivalent to downtime.

If you really need crazy uptime, then you must make sure you have no bugs, because bugs cause more downtime than any hardware failures.

That means an approach to software development that isn't relying on flavour of the month js library.

But we can just stick my flavor of the month js library on lambda and bobs your uncle

who is bob?????

The phrase “Bob’s your uncle” is an idiom, not a specific or placeholder person as per Alice & Bob: https://en.m.wikipedia.org/wiki/Bob's_your_uncle

99.99% of the websites out there don't need 99.99% uptime (from a business point of view)

Once you get to 99.9% uptime you’re still less than 1 hour per month. I used to run apps for companies at work and it was standard operating procedure to have two app servers + load balancer for upgrades.

Drain a server, upgrade it, boot it, check that it works, let it accept traffic. Repeat once more. Done.

It may be true that most businesses don’t need the uptime, but I’d say it’s usually easier to give them high availability rather than field complaints when some random user tries to look something up during a maintenance window.

But where does the data sit? Would you be using some fancy multi-master synchronous replication for that, or is there a database server behind it?

A pair of database servers (leader and hot spare) behind it, although some data was in-app and shared between the two app instances (or more; two was the minimum).

I don’t think of this as particularly fancy or expensive, considering how much you have to pay for engineers and technicians. It was critical enough for >3 nines, and that means having a technician on-call.

I saw this setup at two different companies I worked at, even though the tech stacks were completely different. Whenever I price out setups for similar requirements in cloud I almost always end up with basically the same setup, just on cloud VMs instead of dedicated servers.

Notice that grandparent comment explicitly carved out production servers. If you can't tolerate downtime while you're being upgraded, you're production.

You don't need multiple servers to achieve what you described.

Fine, upgrade the hypervisor, still need two machines.

You don't need two servers rented, even for 99.99% uptime.

That 99.99% uptime is achievable with a single dedicated server, plus cloud as a cold backup and some monitoring to trigger spinning it up.

You will spin up the cloud server when the dedicate server is down. It will cost, but only for brief periods of time.

That's sometimes cheaper than two dedicated servers.

I prefer more than 1 dedicated server, but it's definitely possible to get the 99.99% uptime with just 1.

And in some ways the extra diversity of having a different kind of backup is useful, e.g. for burst scaling as well, and for peace of mind not relying on a single provider.

Even better, you can (and should) run the applications in VMs, or in containers inside VMs. They perform well these days, and even a severe application crash won't take down the host.

Then regular application upgrades don't need a separate host server. Instead you're spinning up and shutting down VMs.

You can also migrate VMs between hosts, and this can be done live if you want. It takes some setting up to have working distributed storage and movable networking, but it's possible.

However, you might decide not to bother with that, as starting and stopping instances and having fleets of instances managed by something like K8S is modern practice for other comforting reasons anyway.

Then the only "big" upgrades are limited to host server upgrades and major network reconfigurations, both of which you will do rarely. You might do host server security upgrades (kernel etc) more often, but that costs the time of a reboot unless it goes wrong.

If those are planned or automated, those "big" upgrades can be done with zero user-facing downtime except in the event of failure, in which case it's limited to the cold-spare startup time

So provided you have the backups in place, maybe on the cloud as a cold-spare so you're not paying for it except when it's needed, you can still have that 99.99% uptime for approximately the cost of one server.

You do not need a separate load balancer device for uptime specifically; only if load balancing is something you want anyway for other reasons. For seamless transitions between backend servers, if you have multiple servers running at a particular time, there are other methods.

On occasions you might instantiate a cloud load balancer temporarily to cover downtime during a planned host upgrade that can't be routed around another way. Like with a cloud cold-spare server, you don't pay for it except when it's temporarily instantiated.

On the flip side, serverless can scale to 0. Makes it very cost effective for one off functions

It doesn't actually scale to 0. The scale to 0 argument is actually just accounting trickery. Servers are still waiting around burning cycles doing nothing waiting to run a serverless function. All that has changed is that Amazon has spread the cost of low utilization across all customers. Which gets baked into your cost as well.

They are good for message queue handlers.

But not good for production REST API endpoints in my experience. In the context of AWS Lambda, you are talking a 29 second timeout in API Gateway. API Gateway is fairly expensive. Also, is connection pooling a solved problem?

Many times you can do better with an EC2 instance running PM2 and Express, or whatever. Like, what does Serverless Framework do locally anyway? Express? Why not just run Express?

I have a production REST API endpoints backed by PHP on Lambda that are called 12k/sec on the high end. no problems since ~2017.

The best part is, I never have to worry about scaling the underlying resources, if sales brings in large clients. Worst case is I ask AWS for a greater Lambda concurrency limit.

Is anyone else worried about the proliferation of disjoint processes that people can whip up using Serverless? At least with a monolith, you've got a relatively coherent grouped together codebase. But with Serverless, you can have some lambda that is infinitely important, but not documented or well known to anyone else in the org.

If you deploy enough Lambdas that depend on each other you effectively end up with a monolith, but the call paths are done through HTTP instead of method dispatch. Arguably this is worse, since discoverability and modification can become extremely difficult.

I think it all depends on how searchable and generally well-maintained the codebase is. Having some obscure critical function hidden in a monolith is totally possible too.

Serverless and microservices aren’t the same thing. You can deploy a monolith to lambda, and make a thousand small services on VMs or orchestration platforms. It’s purely a question of how to organise your code.

I'd say that is misleading at best. Lambdas are inherently unsuited for monoliths due to payload, RAM and execution time limits and cold start times. Lambdas are inherently suited to "real" micro-services, because by nature can only do 1 thing and have strict limits.

Depends entirely on workload - there's 3GB RAM available, which is more than what we use for normal monoliths on Heroku. Payload limits are 6MB or something, which isn't hit for a majority of sites on the internet (each request, not total page size). And there's nothing in the nature of Lambdas that force them to do only one thing. HTTP events can be fed through the normal router (lots of adapters for many web frameworks out there). Non-HTTP events can either go through a switch-case-board. There's no inherent "nature" to lambda, it'll execute whatever code you give it. There are limitations, but those are constantly changing.

I'm ignorant

How does a build trigger the serverless function to notify all of you? I kind of assume serverless means it activates on a timer or when you visit a specific URL. So a build script executes it by visiting a URL when it's finished? And the script executes from any internet connected machine?

Why is it better than having a VPS? I currently use a VPS for a few one off scripts. Cron does the timer ones and the URL ones are entries in a nginx config file. Actually that's half true I actually config an app I wrote to do it cause I didn't want to bother finding out how to do shellexecute on nginx

The serverless framework is an open source tool that you can use to "deploy" "serverless" assets. Basically it covers most of those scripts you usually have to write and gives you a declarative format with some great variable capabilities that help you managed config and declare your components clearly. Running the tool deploys to your cloud of choice and results in functions, APIs, workflows, databases, or whatever you declare now existing and ready for use.

That framework is separate from the class of services regarded as serverless. The star that kicked it off was AWS Lambda. Serverless colloquially means you are dealing with an abstract service contract rather than a server (e.g. the oldest is S3). This removes patching and other maintenance that usually does not directly support business value. More formally, serverless includes auto-scaling to zero, paying only for what you use, high availability, and other design patterns most outside the large tech houses cannot use at low to no cost.

Your VPS is always on and always paid for. It can crash or get in a bad state. It can go out of date and need patching. It is mutable and more vulnerable. It is limited in resources.

Nike reported AWS Lambda scaling in production at 20K RPS/S (0RPS@0s, 20K RPS@1s, 60K@2s, 120K@3s, ...).

serverless-artillery, with loosened account "safety" limits can scale from nothing to producing billions of requests per second on target systems.

Full disclosure: I contributed to the serverless framework and serverless-artillery. I'm a biased fanboy.

Thanks for this concise and pithy description. I keep hearing about serverless...services, but didn’t understand what they were all about.

It’s like your VPS except without having the manage OS and NGINX updates. And probably higher availability if you are concerned about that.

Serverless functions can be set on a timer exactly like cron jobs. But you can also set them to act on certain triggers, when a message comes into an SQS queue, when a notification goes to an SNS topic, etc.

You can do the same with a VPS, but I think VPS cost more because you pay for all the time you're not using them. With serverless functions you only pay for execution time. And with one-off, daily, hourly jobs you end up paying far less, unless your jobs are compute intensive (cron jobs tend to not be...?)

I have seen some VPS these days come in at like $2 a month though, so it's worth comparing the cost.

A 2 man-team with a project manager.

Man1: I'll work on your report.

Man2: I'll watch you work on my report.

That sounds great but if I already have the ability to quickly prototype and stand up one off type services / endpoints in my code base what's the selling point here?

To me it's just full of risk and costs where as adding a new function, getting it reviewed and deploying takes all of a few minutes with none of the cost.

I totally agree with this. My only real complaint is managing these one off items. Team member 1 puts one on a VPC the other doesn’t, another team member ties one to RDS and doesn’t automate auth...etc.

It’s great but you have to put management of usage in place first.

Right, this is where Lambda really does shine. The problem is when it becomes "everything we build has to be serverless"