"So your application produces 60MB of output. The LUNs on the SAN the database server has available is capable of easily sustaining 250MB/s of throughput. Assuming ideal conditions, how fast should it produce the results."
Uh.. about 0.3 seconds?
"Right. However we perform a lot of complex calculations and use some complex logic. Let's assume the penalty for this is two orders of magnitude. How long should it now take?"
About 30 seconds?
"Right. So now do you see why I consider an application that takes 3 1/2 hours to produce the results to have a lot of low-hanging fruit?"
Yeah...
----- -----
I spent the day performing a cursory review. I made three recommendations, only one of which requires non-trivial code changes (but minor in the overall scheme of things), but the overall risk of introducing regressions is very low. Estimated reduction is 97%, which would get it down to about 6 1/2 minutes. Actually refactoring most of the application could certainly get it close to the 30-second mark. The first improvement was simply querying a painful view (which is due to a fundamental schema design flaw) once, instead of querying it N times. It's being tested in their QA department. 82% runtime reduction improvement right off the bat (so from 3 1/2 hours down to 38 minutes)
Not a small company. Very high average level of education amongst the staff. Very bright group in general. Very poor design decisions resulting in very expensive hardware choices to compensate (i.e. FusionIO for a SQL Server tempdb!).
In my experience? Frighteningly real. You could also ask, "What are the chances of finding an application that has quadratic, cubic, or exponential scaling problems served on a silver plate right when he needed it?", and I'd also have the same reply - frighteningly real.
The most fun I've had in my days was not only demonstrating such security holes, but also finding laughably bad scaling issues while I was running amok. The most frightening thing is, I'm not referring to podunk little companies, but wholesalers or manufacturers with 9 and 10-digit annual revenues. And not small internal apps, but their main presence on the web.
Though I'll say, even as a contractor, these gigs are never enjoyable simply because - despite best trying to avoid it - you've managed to embarrass some people on an epic scale. Once you've fixed what you've been paid for, most places don't look to keep you around much longer unless they truly have a "what's right, not who's right" philosophy.