News for hackers. Stuff that matters.
Re:Gor – A simple http traffic replication tool written in Go by StevePerkins on Thursday June 18, 2015 @05:34PM (#9741745)
Eek... that's a rather unfortunate name choice. The "Gor" saga is an infamous series of books spanning the past 50 years, basically the sci-fi/fantasy equivalent of "Fifty Shades of Grey".

https://en.wikipedia.org/wiki/Gor

https://www.google.com/search?q=gor&biw=1920&bih=947&source=...

Re:Gor – A simple http traffic replication tool written in Go by jackmaney on Thursday June 18, 2015 @09:07PM (#9742534)
"There are only two hard things in Computer Science: cache invalidation and naming things."

-- Phil Karlton http://martinfowler.com/bliki/TwoHardThings.html

Re:Gor – A simple http traffic replication tool written in Go by hpkuarg on Thursday June 18, 2015 @10:22PM (#9742708)
Oh, and off-by-one errors.
Re:Gor – A simple http traffic replication tool written in Go by jonathanoliver on Thursday June 18, 2015 @11:18PM (#9742833)
I still count two problems. =)
Re:Gor – A simple http traffic replication tool written in Go by richmarr on Friday June 19, 2015 @05:28AM (#9743750)
No more talk about hard things please. This is a family show.
Re:Gor – A simple http traffic replication tool written in Go by tmd83 on Friday June 19, 2015 @05:02AM (#9743685)
I have always wanted to use such a tool and this one looks good from a cursory glance.

But what I have always wanted to know how people actually use these properly. There are two problems that I see in replicating production traffic to staging/dev

1. Changes in application structure url/parameter. I think given the change we make per release, we will get a lot of error. How to gracefully handle that?

2. Our application is write heavy so there is a lot of new content. So the majority of the requests will access content s that don't exists in the staging/dev environment. We can't even use live replication of DB either since we usually have a lot of DB changes also.

One solution I can think of is record today's traffic. Take a DB snapshot at the end of the day, replicate DB, run migration and then replay. Still has to deal with app change. Am I missing something or this is really challenging. I can see how this will perfectly work though for infrastructure change like server software or configuration.

On a side note of the blue/green deployment. Theoretically the reversible DB migration/parallel deployments of old-new version sounds good. But how many people can make this possible? Does most app has little DB changes from release to release. I will hazard a guess that if we try to implement such things the cost of such complexity will easily overshadow the actual changes to the app and most likely add severe risk of bug. But people seem to be doing it, again not sure what I'm missing.

Re:Gor – A simple http traffic replication tool written in Go by laumars on Thursday June 18, 2015 @06:44PM (#9742066)
Re:Gor – A simple http traffic replication tool written in Go by atom_enger on Thursday June 18, 2015 @09:21PM (#9742580)
We use it to replicate production traffic to our staging and testing environments at https://reverb.com. It's an incredibly useful and functional tool. We've used it to shake out exceptions from our blue/green deploys as well. For instance when we migrated from Rails 3 -> 4, Gor was paramount to shaking out some pretty nasty bugs by replicating prod traffic from the Rails 3 cluster to the new Rails 4 cluster.

Gor is awesome!

Re:Gor – A simple http traffic replication tool written in Go by fweespeech on Thursday June 18, 2015 @04:29PM (#9741438)
This seems like a super useful tool, especially if you want to do blue/green deploys where staging/production end up on the same physical machines and you want to run with artificial load as part of capacity planning/testing.
Re:Gor – A simple http traffic replication tool written in Go by Thaxll on Thursday June 18, 2015 @05:11PM (#9741635)
Looks like Tsung replay feature: http://tsung.erlang-projects.org/
Re:Gor – A simple http traffic replication tool written in Go by mrsirduke on Friday June 19, 2015 @02:07AM (#9743247)
I can't find any documentation, but the name suggests, that Tsung would be doing log replaying.

Gor does packet capturing and duplicates the requests to any number of environments.

It's a different tool for a different task.

Re:Gor – A simple http traffic replication tool written in Go by needcaffeine on Thursday June 18, 2015 @08:39PM (#9742437)
This is super cool. You know what I'd really really like even more? Store this info somewhere, and let me replay it at a later time or date. Maybe I want to replay a whole day's worth of data at an accelerated rate to see if my servers can handle it. It also opens up the ability to do data forensics to try and reproduce a bug, perhaps.

Thanks for this!

Re:Gor – A simple http traffic replication tool written in Go by newman314 on Thursday June 18, 2015 @11:16PM (#9742826)
It appears to have that support.

  # write to file
  gor --input-raw :80 --output-file requests.gor

  # read from file
  gor --input-file requests.gor --output-http "http://staging.com"
Re:Gor – A simple http traffic replication tool written in Go by needcaffeine on Friday June 19, 2015 @01:36PM (#9746386)
Shit, this changes everything.
Re:Gor – A simple http traffic replication tool written in Go by gouggoug on Thursday June 18, 2015 @04:49PM (#9741528)
Seems like an awesome tool.

As a side note, however, I would recommend anybody to not blindly apply the "Tuning" section (at the bottom of the readme) to their system. More specifically, net.ipv4.tcp_tw_recycle and net.ipv4.tcp_tw_reuse are notorious for causing problems if mis-used.

Re:Gor – A simple http traffic replication tool written in Go by oimaz on Thursday June 18, 2015 @05:16PM (#9741653)
Could you elaborate?
Re:Gor – A simple http traffic replication tool written in Go by gouggoug on Thursday June 18, 2015 @09:28PM (#9742595)
Of course,

net.ipv4.tcp_tw_recycle causes problems with NAT-ed clients.

tcp_tw_recycle (Boolean; default: disabled; since Linux 2.4) Enable fast recycling of TIME_WAIT sockets. Enabling this option is not recommended since this causes problems when working with NAT (Network Address Translation).

net.ipv4.tcp_tw_reuse seems fine to use, but literature about its real effects is sparse.

This link http://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-li... mentions both of them.

Re:Gor – A simple http traffic replication tool written in Go by suchitpuri on Friday June 19, 2015 @07:15AM (#9744018)
I have been using VCR , http://www.relishapp.com/vcr/vcr/docs for long time. Though VCR is only applicable to ruby, rails project.

Gor looks promising

Re:Gor – A simple http traffic replication tool written in Go by mrsirduke on Friday June 19, 2015 @02:12AM (#9743258)
I took me a while to find gor, and it seems that there aren't many tools for traffic duplication/replication out there.

If you know of any tools like gor, I'd very much like to know of them.

Re:Gor – A simple http traffic replication tool written in Go by molf on Friday June 19, 2015 @02:50AM (#9743335)
Re:Gor – A simple http traffic replication tool written in Go by zoratu on Friday June 19, 2015 @03:26AM (#9743408)
At the higher end of the performance scale there are commercial options using specialized hardware and commercial software. Building block cost is ~$300k/20Gbps (~400K rps) of replay traffic, but scales out with parallel deployments.
Re:Gor – A simple http traffic replication tool written in Go by LeonidBugaev on Thursday June 18, 2015 @10:59PM (#9742786)
Thanks for putting it to the HN, author here :)
Re:Gor – A simple http traffic replication tool written in Go by room271 on Friday June 19, 2015 @05:00AM (#9743678)
I've used this and it's great. Very easy to use and works.