Re:Show HN: K6 – developer-centric load testing using Go and...
by robbles on Saturday May 06, 2017 @01:55PM
(#14281458)
This looks incredible for work I was previously using Tsung / Gatling for.
One request: please allow the output of either a) a CSV log of every request and it's timing or b) a configurable histogram of request timing, not just some percentiles and averages.
That info is pretty much required for doing any more advanced analysis beyond "the service is fast enough".
https://github.com/tsenart/vegeta does this really well (but doesn't have the scripting abilities of this tool).
Re:Show HN: K6 – developer-centric load testing using Go and...
by robbles on Saturday May 06, 2017 @02:58PM
(#14281696)
That looks like what I'm looking for, thanks! Is there documentation of the format of those JSON logs? CSV would probably be a useful addition just for ease of ingestion into analysis tools and databases.
Once I get a chance to test this out, I'll definitely report any issues through GH.
Re:Show HN: K6 – developer-centric load testing using Go and...
by rlonn on Saturday May 06, 2017 @05:13PM
(#14282292)
I'll update the docs with some info on the JSON format first thing on monday!
If you should happen to try out k6 in the meantime, most of the JSON contents should be recognizable/understandable just by looking at the output, if you know what metrics k6 collects (which is documented on docs.k6.io). You can try running with e.g. 2 VUs for 2 iterations, to get a tiny amount of output data but something that looks like a real load test: k6 run --vus 2 --iterations 2 --output json=output.json script.js
Re:Show HN: K6 – developer-centric load testing using Go and...
by rlonn on Saturday May 06, 2017 @05:04AM
(#14279425)
k6 is our attempt at creating a modern, developer/devops-centric load testing tool suitable for automation. It is released as open source on Github, scriptable in JavaScript (ES6), and built on a Golang core that provides good speed and concurrency/scalability.
The single biggest difference is that k6 lets you write test cases in real code (JavaScript), while Vegeta will only let you supply a static list of URLs to hit. Vegeta has some things k6 does not have, to be fair (like constant RPS-rate load), but k6 was created to address what we saw as missing in Vegeta and most of the other existing tools. We think developers want to write code to test things, so we created a tool that lets you use JavaScript + a scripting API designed for load testing and automation.
But you can actually aggregate results already, if you use InfluxDB as results storage: you can have multiple k6 instances push results data to the same InfluxDB instance. Results are just time series samples, so there are basically no bad effects if you do this - you get all the results in a single db, with correct time stamps. The only difficulty lies in orchestrating the multiple k6 instances so they're in sync and you get the traffic levels you want (i.e. they ramp up/down at the same time). But for a simple ramp configuration, or a config where you run at a stable traffic level, it shouldn't be a problem.
Re:Show HN: K6 – developer-centric load testing using Go and...
by rlonn on Sunday May 07, 2017 @05:52AM
(#14284670)
Thanks, I'll make sure the "params" object gets documented also. Currently, there is not so much you can set per HTTP request. You can basically add HTTP headers and you can also add tags that lets you single out specific results based on those tags (e.g. when using thresholds or when plotting graphs of results).
So - HTTP request timeouts can not be changed at the moment, but this is something I think would be very simple to implement. If you open an issue about it I'll see what we can do to get it implemented quickly.
Re:Show HN: K6 – developer-centric load testing using Go and...
by rlonn on Saturday May 06, 2017 @02:01PM
(#14281488)
No, there is currently no support for websockets. We have been asked for this a couple of times though, and are considering bumping up the priority for a websockets implementation.
As of now, k6 can do standard http, https (and it also supports HTTP/2)
Re:Show HN: K6 – developer-centric load testing using Go and...
by KirinDave on Sunday May 07, 2017 @04:36PM
(#14287511)
I knew it would be unpopular. I get a few u popular posts a year.
But I am very, very irritated by Golang while I am working with it on a daily basis.
It's entire design is informed by assuming that no developer should understand the codebase, but should instead read it over and over. Symmetrically, no insight can be offered in the code, only a rote transcription of an inductive design.
The net result is a language hostile to both those writing and those reading the code. No shortcuts are offered or taken. All error handling must be done manually. All data structures must be reimplemented over and over with different test suites for each one.
One request: please allow the output of either a) a CSV log of every request and it's timing or b) a configurable histogram of request timing, not just some percentiles and averages. That info is pretty much required for doing any more advanced analysis beyond "the service is fast enough". https://github.com/tsenart/vegeta does this really well (but doesn't have the scripting abilities of this tool).
Is JSON useful to you, or do you prefer CSV?
Once I get a chance to test this out, I'll definitely report any issues through GH.
If you should happen to try out k6 in the meantime, most of the JSON contents should be recognizable/understandable just by looking at the output, if you know what metrics k6 collects (which is documented on docs.k6.io). You can try running with e.g. 2 VUs for 2 iterations, to get a tiny amount of output data but something that looks like a real load test: k6 run --vus 2 --iterations 2 --output json=output.json script.js
Had anyone used both to point out the main differences from k6?
The single biggest difference is that k6 lets you write test cases in real code (JavaScript), while Vegeta will only let you supply a static list of URLs to hit. Vegeta has some things k6 does not have, to be fair (like constant RPS-rate load), but k6 was created to address what we saw as missing in Vegeta and most of the other existing tools. We think developers want to write code to test things, so we created a tool that lets you use JavaScript + a scripting API designed for load testing and automation.
But you can actually aggregate results already, if you use InfluxDB as results storage: you can have multiple k6 instances push results data to the same InfluxDB instance. Results are just time series samples, so there are basically no bad effects if you do this - you get all the results in a single db, with correct time stamps. The only difficulty lies in orchestrating the multiple k6 instances so they're in sync and you get the traffic levels you want (i.e. they ramp up/down at the same time). But for a simple ramp configuration, or a config where you run at a stable traffic level, it shouldn't be a problem.
So - HTTP request timeouts can not be changed at the moment, but this is something I think would be very simple to implement. If you open an issue about it I'll see what we can do to get it implemented quickly.
As of now, k6 can do standard http, https (and it also supports HTTP/2)
We'd really appreciate any comments/feedback you may have on exactly how it should be implemented.
But I am very, very irritated by Golang while I am working with it on a daily basis.
It's entire design is informed by assuming that no developer should understand the codebase, but should instead read it over and over. Symmetrically, no insight can be offered in the code, only a rote transcription of an inductive design.
The net result is a language hostile to both those writing and those reading the code. No shortcuts are offered or taken. All error handling must be done manually. All data structures must be reimplemented over and over with different test suites for each one.