Also, hadoop is so painfully slow to develop in it's practically a full employment act for software engineers. I imagine it's similar to early ejb coding.
It's comical how bad Hadoop is compared even to the CM Lisp described in Daniel Hillis' PhD dissertation. How do you devolve all the way from that down to "It's like map/reduce. You get one map and one reduce!"
> They handed me a flash drive with all 600MB of their data on it (not a sample, everything). For reasons I can't understand, they were unhappy when my solution involved pandas.read_csv rather than Hadoop.
User w_t_payne commented:
> I have worked for at least 3 different employers that claimed to be using "Big Data". Only one of them was really telling the truth.
> All of them wanted to feel like they were doing something special.
I think that last line is critical to understanding why a CIO might feel this way.
Do thing starts going like shit as expected now it has to stream all that through the DB bottleneck.
So what's the solution? Well we're in big data territory now apparently at 1.2TiB (comedically small data and almost entirely static data set) and have every vendor licking arse with the CEO and CTO to sell us Hadoop, more DB features and SAN kit.
We don't even need it for processing. Just a big CRUD system. Total Muppets.
Consulting for enterprise customers tends to be a lot like marriages - you can be right, or you can be happy (or paid). It takes a unique customer to have gotten past their cultural dysfunctions to accept responsibilities for their problems and to take legitimate, serious action. But like marriage, there can be great, great upsides when everyone gets on the same page and works towards mutual goals with the spirit of selflessness and growth. Yeah....
"How much data do you expect to have?"
"We don't know, but we want it to scale up to be able to cover the whole market."
"Okay, so let's make some massive overestimates about the size of the market and scope of the problem... and that works out to about 100Mb/sec. That's about the speed at which you can write data to two hard drives. This is small data even in the most absurdly extreme scaling that I can think of. Use postgres."
Even experienced people do not have meaningful intuitions about what things are big or small on modern hardware. Always work out the actual numbers. If you don't know what they are then work out an upper bound. Write all these numbers down and compare them to your measured growth rates. Make your plans based on data. Anything that you've read about in the news is rare or it wouldn't be news, so it is unlikely to be relevant to your problem space.
Not saying this is the case but CIO bashing is all too easy when you're an engineer.
I think the point the author is making is that although they knew from the start that Hadoop wasn't necessary for the job, many people probably don't.
Most of us don't have scaling issues or big data, but that sort of excludes us from using all the fancy new tools that we want to play with. I'm still convinced that most of the stuff I work on at work could be run on SQLite, with designed a bit more careful.
The truth is that most of us will never do anything that couldn't be solved with 10 year old technology. And honestly we should happy, there's a certain comfort in being able to use simple and generally understood tools.
The underlying problem here isn't unique to Hadoop. People who are minimally familiar with how technology works and who are very much into BuzzWords™ will always throw around the wrong tool for the job so they can sound intelligent with a certain segment of the population.
That said, I like seeing how people put together their own CLI-based processing pipelines.