News for hackers. Stuff that matters.
Re:In Head-Hunting, Big Data May Not Be Such a Big Deal by rdouble on Thursday June 20, 2013 @07:03AM (#5911321)
Is Google's HR data set really "Big Data" or just "data?"

Seems like it would fit into a normal database. Or maybe even an unwieldy Excel spreadsheet.

Re:In Head-Hunting, Big Data May Not Be Such a Big Deal by hkmurakami on Thursday June 20, 2013 @07:25AM (#5911391)
I quote my friend who works in "Big Data":

"sometimes I think Big Data is just Excel on 128GB of RAM"

Re:In Head-Hunting, Big Data May Not Be Such a Big Deal by daemonk on Thursday June 20, 2013 @09:37AM (#5911943)
Big data is a misnomer. Complex data is a better description. Having a terabyte of simple data with 2 columns is really not that difficult to analyze and won't give you much information. Whereas having a few hundred mbs data with complex relationships and many dimensions can yield tons of information and is far more difficult to analyze.

Difficulty in "big data" should be about its horizontal breadth (covering many aspects of a system) rather than its vertical depth (covering one aspect of a system in great resolution).

Re:In Head-Hunting, Big Data May Not Be Such a Big Deal by dasil003 on Thursday June 20, 2013 @08:07AM (#5911529)
Not to go all senselessly pedantic, but doesn't Excel have a limit of like 55,000 rows?
Re:In Head-Hunting, Big Data May Not Be Such a Big Deal by willismichael on Thursday June 20, 2013 @09:02AM (#5911774)
The devil is in the details. Big Data is really a massive cluster of VMs running maxed out Excel spreadsheets, and instrumented to restart automatically and restore from redundant backup, a la RAID, when the Excel process crashes one of the Windows VMs.
Re:In Head-Hunting, Big Data May Not Be Such a Big Deal by objclxt on Thursday June 20, 2013 @08:18AM (#5911569)
If you're being pedantic, it's 1,048,576 rows from Excel 2007, 65,536 rows before that.
Re:In Head-Hunting, Big Data May Not Be Such a Big Deal by hkmurakami on Thursday June 20, 2013 @08:49AM (#5911706)
2^20 rows! :)
Re:In Head-Hunting, Big Data May Not Be Such a Big Deal by dagw on Thursday June 20, 2013 @08:23AM (#5911586)
Current limit is 1,000,000 pr worksheet. However there is a tool called PowerPivot which lets you get around that limit and do analysis on larger data sets.
Re:In Head-Hunting, Big Data May Not Be Such a Big Deal by chinpokomon on Thursday June 20, 2013 @11:39AM (#5912709)
I didn't see this response when I replied to the parent. Power Pivot is pretty great when you can use it.
Re:In Head-Hunting, Big Data May Not Be Such a Big Deal by jjcr on Thursday June 20, 2013 @08:32AM (#5911624)
Not since 2007 where it got bumped to over 1 million. http://office.microsoft.com/en-us/excel-help/excel-specifica...
Re:In Head-Hunting, Big Data May Not Be Such a Big Deal by hkmurakami on Thursday June 20, 2013 @08:13AM (#5911550)
He's saying his file has way more rows than that, so maybe they upped the limit in the more recent versions of excel? (I think he also wrote a bunch of VBA and hooked into some external systems too)
Re:In Head-Hunting, Big Data May Not Be Such a Big Deal by chinpokomon on Thursday June 20, 2013 @11:37AM (#5912693)
Not when you use it with Power Pivot.
Re:In Head-Hunting, Big Data May Not Be Such a Big Deal by Joeri on Thursday June 20, 2013 @08:20AM (#5911578)
16 384 columns and 1 048 576 rows actually
Re:In Head-Hunting, Big Data May Not Be Such a Big Deal by n00b101 on Thursday June 20, 2013 @09:36AM (#5911935)
2^14 columns * 2^20 rows * 8 bytes per cell = 128GB. Bang on.
Re:In Head-Hunting, Big Data May Not Be Such a Big Deal by Kurtz79 on Thursday June 20, 2013 @09:14AM (#5911825)
It's just me or this kind of comment is one of the most common in HN lately, when any story about data analysis comes up ?

We get it, big data it's not really "big" unless you talk giga(tera?)bytes.

Do not take upon yourself to educate any single person that misuses the term. It's not worth it. :)

Re:In Head-Hunting, Big Data May Not Be Such a Big Deal by ionforce on Thursday June 20, 2013 @11:51AM (#5912784)
I guess it's a natural reaction to people wanting to jump on that "big data" bandwagon. Kind of like size envy I guess? So sad...
Re:In Head-Hunting, Big Data May Not Be Such a Big Deal by Kurtz79 on Thursday June 20, 2013 @12:56PM (#5913147)
True, the point is that many people writing these stories cannot really tell (or care) about the difference. "Big data" is a sexy definition, so they go with it regardless of wheter it's actually relevant.

Most of the people here do, so these comments are really preaching in the wrong place...

Re:In Head-Hunting, Big Data May Not Be Such a Big Deal by MortenK on Thursday June 20, 2013 @07:36AM (#5911430)
Just data. Working with "big data" is just boring old business intelligence in 99.9% of the cases.
Re:In Head-Hunting, Big Data May Not Be Such a Big Deal by ChuckMcM on Thursday June 20, 2013 @11:04AM (#5912488)
Probably not, but the year that I joined they had processed a million resumes. So they probably have some level of data (ranging from phone screen only to on-site interview) on anywhere from 8 to 12 million engineering candidates. For the folks who have come on site there might be a 5 - 8k words of text in their file for phone screens probably less than 1K depending on if they include a code sample or not. Most of the folks they processed at the time didn't get to on-site interviews so it probably skews to the lower end.

Its "not" big data in the sense that it needs a cluster to process but it is a pretty large sample set of the current population of engineers who might want to work there.

Re:In Head-Hunting, Big Data May Not Be Such a Big Deal by robbiep on Thursday June 20, 2013 @07:08AM (#5911344)
But then you have a tech company that isnt sprucing a buzz word
Re:In Head-Hunting, Big Data May Not Be Such a Big Deal by jedc on Thursday June 20, 2013 @08:45AM (#5911689)
They've reported receiving 1million applications per year. If even a fraction of those get interviewed (with 1-5 interviews per candidate) that's a good chunk of data. Correlate that with regular performance reviews of 30k employees... I'd say that's a small Big Data problem.
Re:In Head-Hunting, Big Data May Not Be Such a Big Deal by raverbashing on Thursday June 20, 2013 @08:52AM (#5911717)
30k? Data. Not Big Data

And "Small Big Data" is probably data as well.

Re:In Head-Hunting, Big Data May Not Be Such a Big Deal by ownagefool on Thursday June 20, 2013 @09:19AM (#5911850)
He's not talking about 30k rows, he's talking about 30k people. It could easily be big data if you monitor & document their every working moment, but they probably aren't doing that so you're probably right.
Re:In Head-Hunting, Big Data May Not Be Such a Big Deal by raverbashing on Thursday June 20, 2013 @09:30AM (#5911908)
Yes, 30k people, so it's what? Some interview reports, some performance reviews, HR report/history of the employee?

It really doesn't look like something big.

Re:In Head-Hunting, Big Data May Not Be Such a Big Deal by jedc on Thursday June 20, 2013 @10:45AM (#5912359)
1 million applications received. Say 10% of those go into some sort of evaluation process = 100k assessments/year. Say 10% of those go through an interview panel of (on average 3 interviews) = 30k assessments/year For 30k employees with (say on average) 2 assessments per year = 60k assessments/year.

So 1 million CVs per year on which to do some sort of evaluations, and 200k individual assessments per year. Over the past five years that roughly 6 million data points.

Since there's no hard-and-fast rule on this, that's why I called it small Big Data.

Re:In Head-Hunting, Big Data May Not Be Such a Big Deal by chris_mahan on Thursday June 20, 2013 @11:09AM (#5912514)
Even if it's 100 million rows. That's something a single beefy server with SQL Server 2012 can handle. That's not big data.

Big data is a million times 100 million rows.

Re:In Head-Hunting, Big Data May Not Be Such a Big Deal by shawabawa3 on Thursday June 20, 2013 @11:19AM (#5912594)
> Big data is a million times 100 million rows.

[citation needed]

This whole thread is pointless. There is no definition of Big data.

Re:In Head-Hunting, Big Data May Not Be Such a Big Deal by raverbashing on Thursday June 20, 2013 @11:57AM (#5912823)
Re:In Head-Hunting, Big Data May Not Be Such a Big Deal by drewvolpe on Thursday June 20, 2013 @03:40PM (#5914399)
Definitely not. And it's a good example of how useful POD ("plain old data") can be. They ask 6 team members 18 questions about what they think of their boss and give those 108 datapoints to her and it's tremendously valuable.
Re:In Head-Hunting, Big Data May Not Be Such a Big Deal by dk8996 on Friday June 21, 2013 @12:46AM (#5916574)
Its "deep learning" data hahaha.