Setting up unit tests is (in my experience) considerably cheaper than setting up integration tests. And it's cheaper to set up a test framework for a project when you've already done so for a similar project, but the time spent learning the required techniques might've put management off allowing time for testing.
For example, consider writing an Eclipse plugin. It's very easy to get a plugin started; there's a template to get you going. But there's no template for a project that will run UI tests with SWTBot. Instead, you have an uphill battle. Create a new project; find the magic incantation to run such tests; realise it's not quite applicable to your project; set up a headless X server on the CI server; learn the test API. It's hard to explain how long all these small things take to someone who's never tried it.
The effort is certainly worth it, but it can be very difficult to justify the spending time on familiarisation and integration of a test framework. It depends on the company, of course. It's good to work somewhere where testing is considered part of software development. Others are more short-sighted.
I think lack of testing is part of a more general problem of not wanting to invest in improving skills and processes, because it doesn't align with short-term goals.
One of the biggest benefits of testing in my mind is improving the design of code. If you have code that is very difficult to test, there is likely something wrong in your design.
Any way you cut it, you need to become knowledgeable about testing to be able to apply it effectively.
And that's just for the back-end processing (nearly 300k lines of C/C++ code). Unit testing? Okay, for large values of "unit", and most of the "units" being tested require almost as much set up as the entire "program".
Is something wrong with the design? Given the constraints and how the project evolved, I can't see it being any simpler. And I'm somewhat overwhelmed with the thought of testing the frontend (which requires Android phones).
You're in a situation with a lot of legacy code. Testing shapes design, but it sounds like you're trying to retrofit testability onto an existing mess. People cut corners for years, and now it's your problem. That sucks.
In your shoes I'd either start improving it or find a new job. I think life's too short to spend my time doing something a computer could and should be doing.
And for the most part, I was able to get the regression test for the backend process(es) to run unattended once started (thankfully---I (along with two others) did it once manually and it was horrible). I have no idea of how to do that for the frontend Android cell phone client. Sure, we can run tests on an emulator, but there are issues with the Android emulator (it exhibits different buggy behavior than the the physical hardware) so that only gets you so far. It's an interesting (if somewhat overwhelming) problem.
Regarding Android, Netflix has a nice article about what they do to test everything: http://techblog.netflix.com/2012/03/testing-netflix-on-andro...
Personally, most of my testing code is unit tests. I try to isolate my code from third-party services in a variety of ways. I do have some integration tests that check that it all works together, but those are never as useful or as maintainable as I'd like.
And to test that other part? Well, it requires I mock out the previous unit (or run it), plus three three other parts (one including a cell phone---which is really a simple script at this point). And again, it doesn't really make sense to test individual routines because this takes the translated IP packets from the SS7 module, and makes several queries to other IP based services. So a lot of what's going on is just simple translations (in a multithreaded/multiprocessor environment---more fun!).
Once I got in the habit of writing tests of even the most simple transformations, the code complexity and my test complexity grew at the same rate, so it's much harder to end up with a giant untestable mass.
It turned out I was calling a non-re-entrant function (indirectly) in a signal handler (so technically it was multithreaded) and the crash really depended on one function being interrupted at just the right location by the right signal. That's why it took a month of staring at the code before I found the issue. Individually, every function worked exactly as designed. Together, they all worked except in one odd-ball edge case that varied from system to system (on my development system, the program could run for days before a crash; on the production system it would crash after a few hours). The fix was straightforward once the bug was found, but finding it was a pain.
So please, I would love to know how unit tests would have helped find that bug. Yes, it is possible to write code to hopefully trigger that situation (run the process---run another process that continuously sends signals the program handles) but how long do I run the test for? How do I know it passed?
I only know three ways to detect this sort of bug, and they all suck: 1) get smart people to stare at all of your code 2) brute force as many combinations as possible 3) move the problem into the type system of your language so you can do static analysis of the code
In this case, once I discovered the cause of the bug I would have written a unit test that exposed it, probably a very focused one. Then I would have gone hunting for other missed opportunities to test for this, and I imagine my team would have come up with some sort of general rule for testing signal handlers.
Oh well, at least in our case our signals originate in IP and we only have to check against a HLR, which do have a semidecent mockup.
And in our case, our signals originate in SS7 ...
receive a message via SS7 and convert it to an IP based message. To test the portion that talks to the telephony network requires a telephony network
I worked on an SMS anti spam/fraud system for a few years and we unit tested and simulated everything.
For unit testing we mocked all the network/hardware stuff so that each part of our code could be tested in isolation. I firmly believe that there is no code which cannot be unit tested[1], though obviously some code is easier to unit test than other code.
For more end-to-end simulation, we wrote a test suite that would simulate the SS7 network and allow us to test our system under all kinds of message flows - testing not just that the system worked for each variant of the message flows, but also stress testing and performance testing our system. It worked with raw SS7 messages received from a number of commercial gateways and also with SIGTRAN messages (which are almost the same thing anyway). This worked pretty well for us.
just simple translations
That should be the easiest type of code to test! Pure functional translation is ideal for testing: if I put in X, I expect to get Y back (for a bunch of X/Y pairs).
You mention multiple machines and multithreading - obviously this makes testing pretty damn hard (though unit testing should generally not be too affected), but possibly also more critical since multiprocessing is hard anyway. Anyway, like I said, I don't know your system.
most of the "units" being tested require almost as much set up as the entire "program"
It sounds to me that the design isn't modular enough (by design or by evolution), or the units are much much too large. Each unit should be fairly simple and reasonably self-contained.
[1] Nowadays I do some embedded systems stuff, which at first I considered really hard to unit test, but changed my mind after reading this book: http://pragprog.com/book/jgade/test-driven-development-for-e... If you can abstract away microcontrollers and other hardware for the purpose of testing in an embedded scenario, you can abstract pretty much anything away.
A boy turns up to school half an hour late, out of breath as he was running into the schoolyard pushing his bike, "Sorry miss, it took me 45 minutes to run here" he explained to the schoolteacher, "Why are you pushing your bike, why didnt you cycle?" the schoolteacher asks, "I was already late when I left the house, I didnt have time to get on my bike"
As mentioned elsewhere, turning testing and code coverage into a dogmatic religion is obviously a bad idea, but when I talk to people about testing it seems that we error hugely on the side of not testing, when you dont think something is testable, it is usually because you didnt design it to be testable from the outset. There is definitely no easier programming guide I have found than a little light that goes green when I have done the right thing, If I have tested code properly it is an order of magnitude less likely to take on technical debt, huge sweeping refactorings are no longer big scary tasks