News for hackers. Stuff that matters.

A comment on Command-line Tools can be 235x Faster than a Hadoop Cluster (2014). Parent comment

Re:Command-line Tools can be 235x Faster than a Hadoop Clust... by geezerjay on Thursday May 24, 2018 @12:19PM (#17145514)
> Now I see people who think 50Gb is “big data” because Pandas on a laptop chokes on it.

IIRC the term "big data" was coined to refer to data volumes that are too large for exiting applications to process without having to rethink how the data and/or the application was deployed and organized. Thus, although the laptop reference is on point, large enough data volumes that make Pandas choke in a desktop environment does fit the definition of big data.

[ Parent | on HN ]
Re:Command-line Tools can be 235x Faster than a Hadoop Clust... by samuel_ms on Thursday May 24, 2018 @02:58PM (#17147147)
That definition might be a bit loose. If you have an existing application designed to run on a Raspberry Pi, it'll hit that limit pretty quickly.

This is obviously an over-exaggeration, but I don't think a dataset that breaks Pandas in a desktop environment even comes close to big data.

[ Parent | on HN ]
Re:Command-line Tools can be 235x Faster than a Hadoop Clust... by gaius on Thursday May 24, 2018 @01:53PM (#17146488)
large enough data volumes that make Pandas choke in a desktop environment does fit the definition of big data

What about too large for Excel, is that “big data” too?

(Neither are)

[ Parent | on HN ]