Re:Stormforger – Cloud-based Load Testing as a Service
by lucas_dohmen on Friday June 20, 2014 @11:28AM
(#7921106)
Wow, this sounds really impressive, especially:
> Here are a few numbers: StormForger did a total of 1,213,583,187 requests in over 50 load test runs to the Quizduell system and a total of about 2.21 TB of data was moved. The error rate was at about 0.000000216% (1 error every 4,624,616 requests).
Re:Stormforger – Cloud-based Load Testing as a Service
by peterwwillis on Friday June 20, 2014 @12:37PM
(#7921451)
Sounds useful for doing end-to-end from a cloud provider to another cloud provider, but there's more lessons to learn when you have 20 different ISPs and a couple dozen different clients sending you traffic in different ways.
For example, for free services without customer support, usually you don't notice the bugs from all these different devices trying to access your service under load. Once you start charging and provide support, suddenly people are calling you with all kinds of edge cases that affect a minority of users, and you have to invest time in tracking them down or refund the money (which is a bad idea since that builds a reputation that you'd rather dump customers than help them).
Re:Stormforger – Cloud-based Load Testing as a Service
by syntern on Friday June 20, 2014 @11:54AM
(#7921236)
Impressive. It's not their fault and they seem to be good and talented, but I can't stop thinking about a possible marketing line: "DDoS as a Service".
Re:Stormforger – Cloud-based Load Testing as a Service
by tisba on Friday June 20, 2014 @04:31PM
(#7922727)
this is especially challenging actually. just because you own the domain name, doesn't mean you own the infrastructure you are pointing your DNS records to!
So adding DNS records is pretty useless to provide proof that you own the resources under test.
Re:Stormforger – Cloud-based Load Testing as a Service
by dyeje on Friday June 20, 2014 @04:39PM
(#7922759)
Something I have been really interested in is being able to replay the actual traffic our site is experiencing. So, we press record and everything that's going to our server is recorded for playback at a later date. Does anyone know any solutions like that?
Re:Stormforger – Cloud-based Load Testing as a Service
by tisba on Saturday June 21, 2014 @04:32AM
(#7924494)
There are a couple solutions that do that - none of them I find sufficient. If you want to clone live traffic, that's probably okay, but recording traffic and replaying it in a usable fashion is a non-trivial task. e.g. systems tend to be dynamic so your recordings would need to be dynamic too in order to be realistic.
Re:Stormforger – Cloud-based Load Testing as a Service
by mping on Friday June 20, 2014 @02:10PM
(#7921942)
The api looks alot like tsung. I'm almost willing to bet that either they got inspiration from it or are using it on the server. Anyway, tsung is great and these services are great too.
Re:Stormforger – Cloud-based Load Testing as a Service
by felixgallo on Friday June 20, 2014 @09:03PM
(#7923728)
if you look at the 'your data - your results' piece of the carousel, it has a log which has 'pid' as a column and, e.g., '<7038,69,0>' as a value. So you're probably quite right.
Re:Stormforger – Cloud-based Load Testing as a Service
by lfuller on Friday June 20, 2014 @05:37PM
(#7923040)
I can't recommend Redline13 (https://www.redline13.com) highly enough. They act as a service that coordinates load testing using instances within your own AWS account.
Really easy to use, and I can run tests that used to cost upwards of $5,000 for under $5. I don't know how they aren't better known amongst DevOps types.
Re:Stormforger – Cloud-based Load Testing as a Service
by hnbascht on Friday June 20, 2014 @01:43PM
(#7921806)
I saw basti's talk on stormforger at last month's interactive cologne – and I think he switched to JavaScript for the test definitions. Maybe the website is a bit outdated. :)
Re:Stormforger – Cloud-based Load Testing as a Service
by carlosdp on Friday June 20, 2014 @02:09PM
(#7921933)
Heroku started with just Rails. It's better to start with one language and do it well and then expand than half-ass it with everything from JS to Erlang imo. The important part is the quality of the service and the information it provides, not the DSL.
Re:Stormforger – Cloud-based Load Testing as a Service
by yoshida on Friday June 20, 2014 @04:18PM
(#7922653)
WRK is a nice tool that can bring you quite far, for sure. But what about the orchestration of complex sessions? As I understood that blog post mentioned in another comment StormForger was actually able to play a game. A simple one, but there is still a lot of complexity involved.
But even if you can build something like this with WRK (which should be possible, due to the Lua support) you still need to setup your test environment (reproducibly), collect all the logs from the machines involved, monitor the test generators, create useful metrics from the test results and make it possible to interact with them. And I'm pretty sure I forget a whole bunch of stuff that needs to be done.
And I know, that I don't want to do all this stuff by myself. With a service providing all this, I can concentrate on the real work: Interpreting numbers and fixing problems ;-)
BTW: I'm pretty sure under the hood there is some load testing tool deployed. Cannot imagine someone runs such a large scale test without such thing.
> Here are a few numbers: StormForger did a total of 1,213,583,187 requests in over 50 load test runs to the Quizduell system and a total of about 2.21 TB of data was moved. The error rate was at about 0.000000216% (1 error every 4,624,616 requests).
For example, for free services without customer support, usually you don't notice the bugs from all these different devices trying to access your service under load. Once you start charging and provide support, suddenly people are calling you with all kinds of edge cases that affect a minority of users, and you have to invest time in tracking them down or refund the money (which is a bad idea since that builds a reputation that you'd rather dump customers than help them).
I know of loader.io but I am specifically looking for ones which I can script to simulate real users.
So adding DNS records is pretty useless to provide proof that you own the resources under test.
Really easy to use, and I can run tests that used to cost upwards of $5,000 for under $5. I don't know how they aren't better known amongst DevOps types.
But why tie yourself to Ruby? (My initial impression is "gross!", but I can't say I'm much of a language purist as I spend most of my time in JS)
Would services like Stripe, Mailgun and Twilio have succeeded if they had forced their customers to use a Ruby DSL?
But even if you can build something like this with WRK (which should be possible, due to the Lua support) you still need to setup your test environment (reproducibly), collect all the logs from the machines involved, monitor the test generators, create useful metrics from the test results and make it possible to interact with them. And I'm pretty sure I forget a whole bunch of stuff that needs to be done.
And I know, that I don't want to do all this stuff by myself. With a service providing all this, I can concentrate on the real work: Interpreting numbers and fixing problems ;-)
BTW: I'm pretty sure under the hood there is some load testing tool deployed. Cannot imagine someone runs such a large scale test without such thing.