News for hackers. Stuff that matters.
Re:Siege – HTTP load tester and benchmarking utility by Veratyr on Monday December 07, 2015 @10:07PM (#10694518)
Siege is great! Really nice and easy to use.

Also noteworthy is Tsung[0], which is (far) more complicated but can speak more protocols, be configured to emulate user behaviour a little more closely and send crazy amounts of traffic.

[0]: http://tsung.erlang-projects.org/

Re:Siege – HTTP load tester and benchmarking utility by robbles on Monday December 07, 2015 @10:55PM (#10694686)
Tsung also has the interesting property of modeling traffic as individual user sessions, with probabilistic thinktimes and dynamic variables. Makes it possible to test your app in a much more realistic way than most tools, which just benchmark a series of identical requests or replay a log.
Re:Siege – HTTP load tester and benchmarking utility by AYBABTME on Tuesday December 08, 2015 @07:25AM (#10695953)
As far as I can tell, siege/vegeta/ab/boom/wrk are all affected by Coordinated Omission, which makes their results of little value. The only tool I'm aware of doing this properly is wrk2. I'm not sure about the distributed tools, free JMeter or hosted like gatling.io and blitz.io, but I'm fairly certain that they all suffer from the same problem.

If you're reaching for a tool like that and can code in Go, implementing a correct one is very easy (~200 loc): just spin up new goroutines for each request at a regular interval.

Re:Siege – HTTP load tester and benchmarking utility by elithrar on Tuesday December 08, 2015 @07:40AM (#10695999)
Gil Tene, the author of wrk2, has a fantastic write-up on the assumptions that lead to Coordinated Omission (and how to avoid them) here: https://groups.google.com/forum/#!msg/mechanical-sympathy/ic...
Re:Siege – HTTP load tester and benchmarking utility by robbles on Tuesday December 08, 2015 @12:22PM (#10697815)
If you write your own, how do you suggest avoiding the same coordinated omission error as those tools?
Re:Siege – HTTP load tester and benchmarking utility by AYBABTME on Tuesday December 08, 2015 @01:01PM (#10698098)
Those tools all make use of a pool of workers. Say you want 1000 RPS, they'll make something like 100 workers and make them do 10 RPS each. To make 10 RPS each, their requests all must take at most 100ms each. The issue of coordinated omission arises when those requests take more than 100ms each and the rate of requests your measuring becomes distorted. This is well explained if you read Gil Tene:

https://groups.google.com/forum/#!msg/mechanical-sympathy/ic...

How to avoid this error yourself? Don't make a pool of workers, have each 1000 requests be their own thread/goroutine. So you'll create 1000 goroutines per second, each of them doing their own request. This way you won't omit measurements by skipping beats.

Re:Siege – HTTP load tester and benchmarking utility by robbles on Tuesday December 08, 2015 @03:57PM (#10699449)
great explanation, thanks!
Re:Siege – HTTP load tester and benchmarking utility by tobz on Tuesday December 08, 2015 @01:03PM (#10698115)
The key is that the request rate be independent of the latency of individual requests.

In the case of wrk (the original wrk, not Gil Tene's version), you could use this version to check for maximum RPS / bandwidth. This is because it's sending requests as fast as it can, but one after the other.

In the case of wrk2, Gil Tene's version, you have to specify the rate at which to send requests. This is how it avoids coordinated omission.

Re:Siege – HTTP load tester and benchmarking utility by anonfunction on Tuesday December 08, 2015 @12:31AM (#10694986)
I've used siege for a long time and never had any problems.

However, I'm easily tempted by the new and shiny which led me to find another great HTTP benchmarking tool: https://github.com/tsenart/vegeta

Re:Siege – HTTP load tester and benchmarking utility by voltagex_ on Tuesday December 08, 2015 @12:49AM (#10695036)
How do siege and vegeta compare to apachebench?
Re:Siege – HTTP load tester and benchmarking utility by anonfunction on Tuesday December 08, 2015 @01:04AM (#10695063)
Siege is very comparable but vegeta has a lot more output options such as graphs, JSON, CSV, etc...

Another tool I forgot to mention was wrk (and now wrk2) which are unique in that they are scriptable via lua.

Re:Siege – HTTP load tester and benchmarking utility by tobz on Tuesday December 08, 2015 @10:50AM (#10697044)
As a small follow-on, there is also Barrage [1], which is more of a framework than anything else. kellabyte wrote it to benchmark changes to Haywire [2], so it's geared towards configuring your client and server, and then collecting all of the metrics from both the client side, and the server side, and displaying them in an easy-to-digest format.

This obviously isn't terribly useful if you're just trying to gauge request rates, or apply load, but it's very useful if you need a repeatable way to not only measure request rates but see how your server is doing under load.

[1] https://github.com/kellabyte/barrage

[2] https://github.com/kellabyte/Haywire

Re:Siege – HTTP load tester and benchmarking utility by petard on Tuesday December 08, 2015 @03:49AM (#10695422)
only used vegeta and ab and prefer vegeta. has many more options and stats, while still being easy to use as ab.
Re:Siege – HTTP load tester and benchmarking utility by VeejayRampay on Tuesday December 08, 2015 @03:05AM (#10695325)
The only problem I've had with siege and similar tools is that they're good to simulate load but from my perspective, they're not as useful when it comes to simulate real-life load, as in multiple IPs, different bandwidths, bursts, etc.

The only services that I found to simulate real life usage were paying services, which is understandable. Maybe in the future with some bittorrent-like load testing framework, I don't know.

Re:Siege – HTTP load tester and benchmarking utility by dalyons on Tuesday December 08, 2015 @03:37AM (#10695399)
ive had a lot of success with Locust. It makes it super easy to simulate actual realistic multi-step user flows with state (eg sign up, then create a group, then add some participants etc), and then to probabilistically weight the frequency of each flow (match it up with how freqently your users do particlar actions). if you do this for say your 3-5 most common user flows, we found you can get a pretty realistic simulation of real load. You can also distribute the "locusts" over a cloud of servers, coordinated via ZMQ.

big step up over seige/AB imho

Re:Siege – HTTP load tester and benchmarking utility by babo on Tuesday December 08, 2015 @04:17AM (#10695486)
I suggest gatling.io as an alternative where you can record and replay browsing session while doing load testing. The tool is pretty interesting, acts as a proxy to record a browsing session and turns it into a Scala DSL code what you can modify to create close to real life scenarios.
Re:Siege – HTTP load tester and benchmarking utility by jespi on Tuesday December 08, 2015 @06:03AM (#10695747)
I know it is not the most popular option, but if it has to be free you can set up JMeter to run with multiple load generator instances. Put those up in a bunch of AWS regions and you have something closer to real life.
Re:Siege – HTTP load tester and benchmarking utility by jfindley on Tuesday December 08, 2015 @04:40AM (#10695539)
For very high traffic loads, I have yet to find anything that compares to wrk (https://github.com/giltene/wrk2). IME, it scales better and more efficiently than anything else I've tried, short of complex distributed tools that are only relevant to a small number of people.
Re:Siege – HTTP load tester and benchmarking utility by busterc on Monday December 07, 2015 @11:05PM (#10694723)
Re:Siege – HTTP load tester and benchmarking utility by mitchtbaum on Tuesday December 08, 2015 @12:18AM (#10694957)
Re:Siege – HTTP load tester and benchmarking utility by hamilyon2 on Tuesday December 08, 2015 @04:20AM (#10695496)
I am surprised nobody mentioned tank (https://github.com/yandex/yandex-tank) yet. I found it highly configurable, relatively easy to use. For those who is interested in good perfomance and features, it might be a good choice.
Re:Siege – HTTP load tester and benchmarking utility by siscia on Tuesday December 08, 2015 @04:24AM (#10695506)
I am using siege to test my last project, Numerino[1] a priority queue, my experience has been great, easy to use and very performant.

[1]: https://github.com/siscia/numerino

Re:Siege – HTTP load tester and benchmarking utility by zeisss on Tuesday December 08, 2015 @06:03AM (#10695746)
and there is also Gatling.io ;) http://gatling.io/