Re:Improving testing by using real traffic from production
by lazyjones on Wednesday June 05, 2013 @05:20AM
(#5824447)
Good idea to use tcpdump in order to record the requests as they happen (not as they're processed, i.e. from the logs as other replay tools), but perhaps using it in promiscuous mode is unnecessary if the listener runs on production servers. Oh, and bonus points for using Go.
Re:Improving testing by using real traffic from production
by lawl on Wednesday June 05, 2013 @09:14AM
(#5825306)
This is very cool. Could have needed than in my last project where in our load-testing with j-meter everything was fine and in production our nodes would fly out of the cluster like flies.
I haven't looked deeply at it but I think session support would be very important, so that you don't just forward the requests but take care to maybe rewrite the session cookie to the server assigned one and when rate-limiting make sure to prefer to keep existing user sessions. From my expirience the problems we had on production were often because of some weird steps user took, which you might not catch if you don't make sure to exactly keep the users sessions in sync.
Re:Improving testing by using real traffic from production
by LeonidBugaev on Wednesday June 05, 2013 @09:19AM
(#5825333)
Yeah, this version have lack of smart rate limiting. But i want to implement "buffering", so requests will be send in same order, without loss, from buffer. Smth like video streaming. Session loss in this case will be only when i need to re-fill buffer.
Re:Improving testing by using real traffic from production
by lawl on Wednesday June 05, 2013 @09:42AM
(#5825465)
Or when the session times out on the server side because it's rate limited. I'm just talking about my own use case I would have had. We had a session timeout of 10 minutes there. But i realize this might be an edge case.
Re:Improving testing by using real traffic from production
by sparrovv_ on Wednesday June 12, 2013 @06:44PM
(#5871201)
You wrote that staging should be the same as production env (and I agree), so how does it help with catching new errors if they were already caught and reported on production?
Of course we could run that against our test environment, but then I'd be afraid of getting to many errors, that were caused by changes that we want to release.
Re:Improving testing by using real traffic from production
by kohlerm on Thursday June 06, 2013 @03:13AM
(#5830952)
Is your application completely stateless?
I'm asking because Load test tools, which are based on http replaying always support rewriting rules, because in stateful web applications the server may parse a cookie (or something similiar) to the browser, which is then used in the next requests.
Re:Improving testing by using real traffic from production
by jrochkind1 on Wednesday June 05, 2013 @08:09AM
(#5825031)
Nice! This is something I've wanted to do for a while, but whenever I started down the path of hacking together my own tool, ended up giving up after a bit of work realizing it was a bit more complicated to do well than I anticipated and I didn't have time to do it right.
Re:Improving testing by using real traffic from production
by jondot on Wednesday June 05, 2013 @07:52AM
(#5824965)
Great tool. I was just about to do such a thing - thanks!
I actually planned on a reverse proxy in Go, but seeing now that you've implemented with tcpdump - any insights on the two ways to do this?
Re:Improving testing by using real traffic from production
by a-priori on Wednesday June 05, 2013 @09:01AM
(#5825241)
I think tcpdump is the better solution since it's less intrusive into your production environment. Even if the tool suddenly crashes, it will not interrupt the flow of requests to your servers.
Something similar, like using libnetfilter_queue, would also be good.
Keep up the good work.
Of course we could run that against our test environment, but then I'd be afraid of getting to many errors, that were caused by changes that we want to release.
So is there something obvious that I'm missing?
Something similar, like using libnetfilter_queue, would also be good.
Site hosted using Github Pages.