News for hackers. Stuff that matters.
Re:35M Hot Dogs: Benchmarking Caddy vs. Nginx by pdhborges on Friday September 16, 2022 @11:22AM (#32867314)
The author linked to wrk2 but I think he ended up using a k6 executor that exhibits the problem wrk2 was designed to solve.
Re:35M Hot Dogs: Benchmarking Caddy vs. Nginx by hassy on Friday September 16, 2022 @11:43AM (#32867684)
Yep, k6 suffers from coordinated omission [1] with its default settings.

A tool that can send a request at a constant rate i.e. wrk2 or Vegeta [2] is a much better fit for this type of a performance test.

1. https://www.scylladb.com/2021/04/22/on-coordinated-omission/

2. https://github.com/tsenart/vegeta

Re:35M Hot Dogs: Benchmarking Caddy vs. Nginx by imiric on Friday September 16, 2022 @06:37PM (#32872925)
With its default settings, yes, but k6 can be configured to use an executor that implements the open model[1].

See more discussion here[2].

[1]: https://k6.io/docs/using-k6/scenarios/arrival-rate/

[2]: https://community.k6.io/t/is-k6-safe-from-the-coordinated-om...

Re:35M Hot Dogs: Benchmarking Caddy vs. Nginx by hassy on Saturday September 17, 2022 @07:57AM (#32876783)
Yes indeed but wrk2 or Vegeta is still better for this particular use case (unless k6 has support for setting a constant RPS rate, afaik it does not), as otherwise the overhead of establishing a new TCP connection for a single HTTP request will dominate the benchmark.
Re:35M Hot Dogs: Benchmarking Caddy vs. Nginx by imiric on Sunday September 18, 2022 @01:14PM (#32889332)
> unless k6 has support for setting a constant RPS rate, afaik it does not

Yes it can, via the `constant-arrival-rate` executor[1].

> the overhead of establishing a new TCP connection for a single HTTP request will dominate the benchmark

By default, k6 will reuse TCP connections, and you have to explicitly disable it[2].

I'm not saying that wrk2 or Vegeta wouldn't be a good fit for this test, but k6 is also capable of it, with some minor configuration changes.

[1]: https://k6.io/docs/using-k6/scenarios/executors/constant-arr...

[2]: https://k6.io/docs/using-k6/k6-options/reference#no-connecti...

Re:35M Hot Dogs: Benchmarking Caddy vs. Nginx by tylerjl on Friday September 16, 2022 @02:33PM (#32870223)
Damn. This is probably worth swapping out k6 for if I manage to pull off a second set of benchmarks. Thanks for the heads-up.