Warm vs cold start is not a very meaningful comparison, though. Lambda typically scales from zero. Any warm system is going to greatly outperform a cold one. Java on Lambda can vary from nanoseconds to seconds, depending on whether you hit a warm copy or a cold start.

Disclosure: I work on a different FaaS, Project Riff.

I benchmarked all the frameworks in cold & warm, Warm is also very slow, no concurrency etc'. how many events/sec can u do with a warm Riff instance e.g. in Go & Py? whats the latency?

Let me put it another way: are you saying you have no cold-start penalty?

Edit: and to answer your question -- no, I have not personally benchmarked it. Can you point me to your suite? I've been thinking it would be useful to work on a benchmark that different implementations can share to catch regressions (and earn bragging rights, of course).

Great idea, we suggested to create an open benchmark at CNCF, a bunch of members opposed, we can come with a joint proposal. You can use our test tool https://github.com/v3io/http_blaster, it can generate a workload of > 600K http req/sec with one client and is programable.

Do you have a link to the benchmarking discussion? I'd be interested to see what the objections were.

I raised the topic today in a team meeting and I'll probably get to tinker over the next few weeks. I'm interested in exploring a few different axes of stress. RPS is one, but also things like "how does it behave with ten thousand functions installed?", "how does it behave with traffic yo-yoing?" etc.