>By definition your predictions will be more accurate the smaller the window is.

Small nit, and this doesn't detract from your points. I don't think this is universally true by definition, even if it is almost always true. You could come up with some rare conditions where your traffic at t+5 minutes is actually easier to predict than at t+20 seconds. Of course, even in that case you're better off (ceteris paribus) being able to spin things up in 20 seconds.

I can come up with a lot of examples where it is easier to predict further out[0], but that also means I can predict them 20 seconds out. :)

[0] For example I can tell you exactly when spikes will happen to Netflix's servers on Saturday morning (because the kids all get up at the same time). And I can tell you there will be spikes on the hour during prime time as people shift from linear TV to streaming (or at least they did a lot more 10 years ago!). I can also tell you when spikes to Alexa will be because I already know what times peoples alarms are set for.

Hm... I think there are things with one sided uncertainty, that make the ops example possible.

For example, things rarely start before they're scheduled, so if you know something is supposed to happen at 5 and last an hour, it's reasonable to assume it will be happening at 5:10, so if it's 4:59 right now, you're more certain that it will happen in 11 minutes than in one minute.

Of course, as time progresses your uncertainty curve flattens out (at 5:09 you're pretty certain it'll happen any moment now), but that's only in the future.

Even then, it's quite reasonable to say that there's no time leading up to the event where you're more confident it will start imminently than you're confident it will be occurring further out.

won't there be more noise while predicting just 20s in advance? The longer the duration, the less effects we will see of temporary events like network blips etc. no? sorry I'm new to software engineering and just trying to learn.

There’s no one answer for it, you need to learn your traffic / resource usage patterns and tune the scaling to match your situation.

No shortcuts really, although a lot of web applications behave “kinda” similar.

Start conservatively and tweak from there.

Faster feedback is the shortcut. And it always works. Faster boot time is lower latency to serve the queue. Faster feedback is stabilising.

However with a smaller prediction interval you can dampen your autoscaling more. If you predict 20s into the future, react, and 20s later you see how that changed the situation you can afford to spin very few instances up and down each 20s. If you have to predict 5m into the future you might have to take much stronger actions because any effect is delayed by the 5m startup interval.

Fully agree, doing reactive autoscaling when the actual boot time is slow is an inherently hard problem. We've done years of research into building specialized VMs (unikernels) and fast controllers to be able to provide infra that allows VMs/containers to cold start, and thus autoscale/scale to zero in milliseconds (eg, a simple Node app cold starts in ~50 ms). If interested, you can try it out at kraft.cloud, or check out info about the tech in our blogs (https://unikraft.io/blog/) or the corresponding LF OSS project (www.unikraft.org).

Unikraft is really cool, but Linux is not necessarily the blocker. You can boot to PID1 in firecracker in ~6ms, see my experiments: https://blog.davidv.dev/minimizing-linux-boot-times.html

Autoscaling also makes performance insights easier. It keeps the resources per processed request relatively consistent over time. Whereas resizing not automatically can leads to a lot of operational complexity in understanding how your service will react under different loads.

This dang age everyone should use autoscaling. relying on ODCR (capacity reservations) for guaranteeing resources exist.

I assume the majority of people are autoscaling merely for diurnal and weekly traffic swings, where the signal window could be as high as 30 min or 1 hour. Do folks really see sub-minutely autoscaling?

Are customers that are so cost sensitive served well by public clouds and attendant infrastructure?

Netflix scales on short intervals for reliability. Scale up quickly to handle a spike in traffic, scale down slowly to save money.

They also do proactive scaling to address the issues you brought up, since they can predict with fairly high accuracy the normal viewing patterns.

For example scaling up on Saturday morning ahead of all the kids waking up.