Re:Load Test Rails4 app ruby 2.1 MRI Vs Rubinius 2.0
by cpuguy83 on Wednesday October 09, 2013 @09:32AM
(#6521089)
This is in no way a good test.
Yes, Rubinius takes longer to load than MRI. It'a also JITed, so it needs time to warm up to start screaming.
Also, this is in dev mode... did you disable Rack::Lock?
MRI can only handle 1 request at a time, Rubinius can handle whatever your concurrency level is (assuming you've removed Rack::Lock from your environment)
Re:Load Test Rails4 app ruby 2.1 MRI Vs Rubinius 2.0
by timdorr on Wednesday October 09, 2013 @09:41AM
(#6521147)
> This is in no way a good test. Yes, Rubinius takes longer to load than MRI. It'a also JITed, so it needs time to warm up to start screaming.
This is a fair point. You should run ab multiple times over a longer duration of time. 5 seconds seems like a short time to declare an absolute winner.
Re:Load Test Rails4 app ruby 2.1 MRI Vs Rubinius 2.0
by jacques_chester on Wednesday October 09, 2013 @10:42AM
(#6521503)
The ACM's SIGMETRICS[1] are the main group publishing this kind of research. Some of it gets very specialised -- just picking a title from the most recent Performance Evaluation Review[2], I get "YOUQMON: a system for on-line monitoring of YouTube QoE in operational 3G networks".
However, being the ACM, it's basically inaccessible unless you sell a kidney.
An organisation with a slightly more industrial focus is the Computer Measurement Group[3]. Like the ACM ... bring money.
Re:Load Test Rails4 app ruby 2.1 MRI Vs Rubinius 2.0
by look_lookatme on Wednesday October 09, 2013 @11:14AM
(#6521720)
Sorry to go off-topic, but does anyone know if it's possible to get your hands on these pubs/journals for free at the Science, Industry and Business Library [1] in NYC?
Re:Load Test Rails4 app ruby 2.1 MRI Vs Rubinius 2.0
by warmwaffles on Wednesday October 09, 2013 @10:27AM
(#6521407)
I would also love to see an application in production measured with NewRelic or something else and compare the results. A good way to test this would be to spin up three servers.
* 1 load balancer that plays round robin on the requests.
* 1 server running MRI
* 1 server running Rubinius
Each with NewRelic installed and compare the A/B test that is taking place for a period of about a week of heavy usage. Obviously the gems that the rubinius application would have would be different than what the MRI gems would be. But the core of the application would remain the same.
This would give me and others better insight as to how the two interpreters are running in a production.
Re:Load Test Rails4 app ruby 2.1 MRI Vs Rubinius 2.0
by bhauer on Wednesday October 09, 2013 @10:33AM
(#6521446)
Although you may be running a single-threaded test intentionally, I generally recommend using a multi-threaded load simulation tool versus Apache's ab tool [1]. My preference is wrk [2], especially since it provides for time-limited tests rather than request-limited. Another alternative is WeigHTTP [3], which is basically a multi-threaded replacement for ab.
Re:Load Test Rails4 app ruby 2.1 MRI Vs Rubinius 2.0
by mrinterweb on Wednesday October 09, 2013 @10:18AM
(#6521354)
Not necessarily. No warmup time was allowed for rubinius's cache. Also we don't know how concurrency was being handled. Rubinius should be able to utilize multiple cores. More information about the hardware, rails app, way that the puma server is being started, warmup time, duration of request time spent on the database, etc should be made available before any remote conclusion like which one is faster can be made.
Re:Load Test Rails4 app ruby 2.1 MRI Vs Rubinius 2.0
by warmwaffles on Wednesday October 09, 2013 @10:24AM
(#6521384)
No. Rubinius' JIT has not had time to warm up. These tests are a little unreliable. I can't find the article off hand, but the way these tests were done aren't exactly telling me anything. Power of ten tests don't do much other than show me that the tester is naive.
Also, this is in dev mode... did you disable Rack::Lock?
MRI can only handle 1 request at a time, Rubinius can handle whatever your concurrency level is (assuming you've removed Rack::Lock from your environment)
This is a fair point. You should run ab multiple times over a longer duration of time. 5 seconds seems like a short time to declare an absolute winner.
This most likely means Rack::Lock is used in your test.
Some questions:
* How is startup time controlled for?
* Why only one run?
* Why so few requests?
* Why no warmup period?
* Why this particular configuration?
* What is the OS, hardware configuration?
* Why is it running in a development environment instead of a test or prod environment?
However, being the ACM, it's basically inaccessible unless you sell a kidney.
An organisation with a slightly more industrial focus is the Computer Measurement Group[3]. Like the ACM ... bring money.
[1] http://www.sigmetrics.org/
[2] http://dl.acm.org/citation.cfm?id=2518025&CFID=252561427&CFT...
[3] http://www.cmg.org/
[1] http://www.nypl.org/locations/sibl
iMac Intel Core 2 Duo 3,06 GHz cpu 1
* 1 load balancer that plays round robin on the requests.
* 1 server running MRI
* 1 server running Rubinius
Each with NewRelic installed and compare the A/B test that is taking place for a period of about a week of heavy usage. Obviously the gems that the rubinius application would have would be different than what the MRI gems would be. But the core of the application would remain the same.
This would give me and others better insight as to how the two interpreters are running in a production.
[1] https://news.ycombinator.com/item?id=6211883
[2] https://github.com/wg/wrk
[3] https://github.com/lighttpd/weighttp
So which is correct?