News for hackers. Stuff that matters.
Re:Unit testing a TCP stack by soheilhy on Thursday July 09, 2015 @11:59AM (#9858673)
Google has a pretty mature TCP testing framework, called packetdrill: http://static.googleusercontent.com/media/research.google.co...

It supports all the features you need, from packet structure to timing: https://code.google.com/p/packetdrill/

Re:Unit testing a TCP stack by acveilleux on Thursday July 09, 2015 @12:56PM (#9859129)
It still treats the whole stack as a blackbox. Necessarily the effort to cover important boundary cases throughout the TCP stack can get overwhelming.
Re:Unit testing a TCP stack by ori_b on Thursday July 09, 2015 @03:18PM (#9860104)
Honestly, I've found black box testing at public boundaries of a system gives the best payoff vs development time, especially if you can make them run quickly.

Most issues seem to be catchable with this, and way too many unit tests end up becoming 'change detector' tests, where you test tautologies about implementation details.

Re:Unit testing a TCP stack by gsnedders on Thursday July 09, 2015 @08:36PM (#9861675)
Black-box testing as a first move is often a good move, IMO — but it's important to accept that at some point you're going to stop hitting so many edge-cases in the implementation that ought be tested. Some of those edge-cases should probably be tested with black-box tests, and some should probably be tested with unit tests.

One has to consider the performance cost of black-box tests, though. It will almost always be higher than that of unit tests, simply because more code will get run per test (this is especially true of network stuff, as even going through the loopback device is comparatively insanely expensive). That said, in some cases, it may well still be the right choice.

Re:Unit testing a TCP stack by golergka on Friday July 10, 2015 @09:04AM (#9863690)
> black box testing at public boundaries of a system

But that sounds more like integration testing, not unit testing, doesn't it?

Re:Unit testing a TCP stack by jsnell on Thursday July 09, 2015 @04:26PM (#9860444)
I watched the Usenix talk on packetdrill, and while it's a really neat tool it seems to me that it doesn't really support timing very well.

The tests are run in real time - running 650 tests take almost half hour, which would normally be considered completely unacceptable for any kind of TDD. The timing can also vary from one run to another, so all timestamps have a few milliseconds of wiggle room automatically, and a small proportion of runs (1/2500) still fail randomly due to timer inaccuracies. And unit tests that fail non-deterministically are really unpalatable. Is the talk outdated re: these issues?

Great hack despite that, and I would certainly have loved to write a scripting language for expressing tests rather than use C++. And the packet syntax they've chosen is probably superior to what I ended up with. But what I describe in the post dates back to 2011, so packetdrill wasn't yet around for us to take inspiration from :-/