News for hackers. Stuff that matters.

A comment on Don't use Hadoop when your data isn't that big (2013). Parent comment

Re:Don't use Hadoop when your data isn't that big (2013) by orisa2 on Tuesday May 23, 2017 @01:23PM (#14403107)
It's interesting that's the connotation that was brought to mind for you. In the context of this thread, Google, Facebook, and Amazon are the definition of enterprise infrastructure. Google laid the ground work for much of the enterprise big data infrastructure. They've also developed buzz worthy software such as kubernetes, and spanner. Facebook and Google both are known for big data machine learning. Amazon brought the cloud to the mainstream, and offer many popular big data services through AWS. In general, there are few companies in the world who operate near the scale, with the reliability of these giants.

I think you're conflating enterprise with old and stuffy, and non-enterprise with bright colors and cutting edge technology. When I think of enterprise I think of software that needs to operate at scale with strict requirements on performance and uptime.

Apple loses a lot of the talent to other companies, and has never really been known having strong technology, so I understand that.

[ Parent | on HN ]
Re:Don't use Hadoop when your data isn't that big (2013) by nostrademons on Tuesday May 23, 2017 @03:03PM (#14404175)
The interesting thing is that Google and Facebook created big-data solutions to solve actual problems they were facing. There are plenty of Google data scientists that reach for R or Pandas well before they write a MapReduce, and if you do need to write a MapReduce (well, Flume/BigQuery now), it's highly recommended to run it on a sampled dataset before extending to the full corpus.

There are some "enterprisey" companies that do the same, but there are also a whole lot of companies that reach for big-data tools because they want to be like Google, ignoring that their problems are actually quite different from the problems Google faces.

[ Parent | on HN ]
Re:Don't use Hadoop when your data isn't that big (2013) by cookiecaper on Tuesday May 23, 2017 @03:17PM (#14404293)
Yes, I strongly believe that this has been one of the strongest drivers of tech fads since at least 2010. People want to be like Google, so they copy Google. They don't understand that Google probably would've loved to be able to make the thing work with Oracle v. spending years developing their own internal systems, but the unique problem space put them at the disadvantage of needing to use a completely custom solution.

Google publishes an academic paper on this and the general public misinterprets it as a recommendation. Soon you see people writing open-source implementations "based on the GoogleThing Paper", and a new tech fad is born. It will consume billions of dollars before it dies in favor of another fad "based on the FacebookThing/TheNextGoogleThing Paper".

Walk up to most business guys and they will jump at the chance to "become more like Google". Try to talk them down from this, and your challenge is to convince that no, we don't want to be more like one of the most important and influential technology companies in the world, the company that's on the news every day, and whose logo he sees every time he looks at his phone, and the company who keeps taking all of the best hires from the universities. Worse, you'll be making that argument because "we're just not as big [read: important] as them". Not a promising position for the reasonable engineer.

This has been a terrible blight on our profession these last several years, but we just have to learn to roll with it. It's only by understanding and accepting the psychology around this that we can formulate effective counterstrategies, or make the best of the situation that's before us.

[ Parent | on HN ]
Re:Don't use Hadoop when your data isn't that big (2013) by threeseed on Tuesday May 23, 2017 @02:32PM (#14403878)
> Apple loses a lot of the talent to other companies, and has never really been known having strong technology, so I understand that.

That statement is just ridiculous.

Apple innovates a lot in the mobile and desktop spaces and on the software side they have pushed a lot of projects forward e.g. WebKit, LLVM. They also run some very large web services e.g. iCloud, Messages which are on par with some of the challenges Google and Facebook have.

[ Parent | on HN ]
Re:Don't use Hadoop when your data isn't that big (2013) by pjmlp on Tuesday May 23, 2017 @02:44PM (#14403977)
You should talk to iOS developers about iCloud.
[ Parent | on HN ]
Re:Don't use Hadoop when your data isn't that big (2013) by threeseed on Tuesday May 23, 2017 @03:11PM (#14404253)
100M people have downloaded apps I wrote. I know all about the issues with iCloud.

But as a web service that underpins so much of iOS it is still on a scale and complexity that rivals anything Google and Facebook has. Apple doesn't get enough credit for actually make this work on a daily basis.

[ Parent | on HN ]
Re:Don't use Hadoop when your data isn't that big (2013) by titanomachy on Wednesday May 24, 2017 @12:56AM (#14407688)
Well, they get hundreds of billions of dollars. I doubt they feel unappreciated.
[ Parent | on HN ]
Re:Don't use Hadoop when your data isn't that big (2013) by zaphar on Wednesday May 24, 2017 @09:12AM (#14409729)
iCloud is impressive and is easily on a scale that most companies will never reach. But Google and Facebook are on an entirely different level. The comparison isn't even close. iCloud isn't a rival. It's more of a distant cousin.

They definitely deserve credit for making it work because even at their scale it's an amazing feat. But there's no comparison to Google or Facebook's scale.

[ Parent | on HN ]