(a) Future analysis techniques are unknown. Today you might be using method X, but tomorrow you might want to try method Y which calls for an entirely different massaging of the raw data.
(b) Backtesting and replay. Simulations and replays of previous trading events are valuable not only for testing a model but also testing the particulars of new "infrastructure" code.
Both require high-fidelity source data. This stuff is so hard to come by and costs so much money that most people want to be safe and save it forever.
To your point on analysis: it is rare that one would directly analyze the source data itself. You'll almost always want to transform the data into something more manageable for model development. The most crude form of this is idea of "bars". Instead of looking at tick data, traders tried to reduce the noise by looking at arbitrary aggregations of that data: 1 minute, 1 hour, 1 day, etc. Technical analysis using moving averages and other indicators are also examples.
Not that I'm a mathematical or economic genius of any kind, but I continue to be surprised at how primitive financial analytics seem. When people do find something interesting (eg Li's Gaussian Copula) they almost invariably make a fetish out of it and hurl themselves off the nearest cliff shortly afterwards. Economics faculties are as much to blame as anyone, I feel.
With respect to your news about a CEO, what he told me is that any stocks that had news or upcoming news were simply pulled off the table from trading. I always thought that there could be some value in parsing and understanding news, but from his standpoint it simply took too long. His other point is why deal with other variables when you don't have to. Find low news stocks and you find something that is more easily predictable.
That leads me to the next point he made, time. I asked him about using exotic analysis techniques and again he said most of their algos were variations of regression testing. Now, he didn't go into how they figured out the variables in their regressions, but he said the reason they stuck with those variations is because they were fast and they allowed them make decisions quickly before the window of profit closed.
Last I heard he retired at 30 traveled the world with his wife got bored and now runs a hedge fund. I don't know if how he was doing it 10 years ago would apply to today.
Perhaps one shouldn't? I've always been a bit fascinated by the analytical tools in brokerage software - with a pretty good understanding of DSP and an appreciation for the fact that asset prices are somewhat periodic, it's hard to overlook the fundamental similarities between stock graphs and audio waveforms. Once you start performing FFTs or wavelet transforms and get a 'feel' for dealing with signals, patterns become very seductive...possibly too seductive: http://en.wikipedia.org/wiki/Pareidolia and http://en.wikipedia.org/wiki/Tetris_effect
Might there be another approach? You wouldn't prepare to go to the store by reviewing and analyzing the 1287 individual footsteps of your previous trip, or try to predict the content of a HN thread by textual analysis of all previous threads. Do we do so at a subconscious level, then? Not really - or rather, our subconscious tends to forget about things as soon as they cease to be important, which in the case of things like walking is a period of seconds or less. Processing large volumes of data is computationally expensive, but it turns out that simple rules can yield results that are both complex and useful, as in flocking and swarming behavior: see http://en.wikipedia.org/wiki/Boids and http://en.wikipedia.org/wiki/Swarm_Intelligence, plus everything from the wisdom of crowds to nonlinear dynamic systems (aka chaotic ones) like Newton's basin or the logistic equation.
I feel there are two other fundamental problems with the massive dataset + analysis approach. One is that you're not working in a closed system, and there's no sensible way to quantify unexpected events. 'Bigcorp CEO in Sex Scandal!' might cause the price of Bigcorp to tank if it's a major distraction or their largest customer base is among rural conservatives. If Bigcorp makes racing cars, it might just be good publicity! Now you can do some kinds of interesting posthoc analysis (eg for news stories that contain a stock symbol, measure the correlation between # of textually similar stories and stock volume/prices using a distributed windowing function) but we're a long way from having a browser plugin that trades based on the contents of your RSS feed.
Another problem is that of feedback. As you've discussed so ably above, people who spot an arbitrage opportunity will mine the hell out of it. And as we all know, traders are extremely subject to herding behavior even though all training suggests they do otherwise. Sure there are systematic contrarians, but I bet that if you just want to do academic analysis you could find a contrarian coefficient and quantify its damping effect on price or volume movements.
So rather than crunching vast quantities of stored data, I wonder if it might be better to treat price movements not as absolutes which you hope will reach a particular ceiling or floor, but as differential vector data with a short half-life. So far AI and modeling approaches seem to have focused on prediction (surprise) and don't perform especially well. I think it would be more interesting to map correlation variations for as large a number of nodes (listed securities) as possible - think how we intuitively appreciate the dynamics of a school of fish when watching a nature documentary, without performing any detailed analysis of individual fish trajectories.
Of course this still involves processing a lot of data, but storing it is less important because are only seeking to become more familiar with high-level behaviors inside that system. There's more to fishing than running trawlers!