News for hackers. Stuff that matters.
Re:Why literature is the ultimate big-data challenge by Isamu on Friday March 03, 2017 @11:16AM (#13783854)
I would be interested in reading some technical papers on the subject, could someone post a few? Whether about Shakespeare or other literature analysis?

That said, this is SO not big data. It is small data, but potentially interesting analysis.

Re:Why literature is the ultimate big-data challenge by NarcolepticFrog on Friday March 03, 2017 @11:58AM (#13784184)
I think it depends on what you mean. I don't think that just the number of bytes of data should be used to measure the size of your dataset. For example, if I have 1TB of all 1's, this is a lot of data, but not very interesting.

I think a more nuanced notion of size is the /information content/ of the datasets. I haven't thought about it carefully, but I'm sure you can quantify this more explicitly in terms of information theory or other non-bit-based complexity measures.

From this point of view, literature may not be large (in terms of the number of bytes), but in terms of information content, it is incredibly dense. A large portion of human knowledge is written somewhere. From this point of view, it is in fact big data.

Re:Why literature is the ultimate big-data challenge by Isamu on Friday March 03, 2017 @12:56PM (#13784680)
Maybe you would find the theory of information interesting:

http://math.harvard.edu/~ctm/home/text/others/shannon/entrop...

Here in his famous tour de force Claude Shannon lays out the way we can estimate the amount of actual information in an act of communication (e.g. a literary work) and relates it to system entropy.

To your point, !TB of all 1's compresses to a just few bits of actual information.

But I suspect you are not actually speaking about the information contained in a literary work. You are probably thinking: how much expansive commentary and explanation could a literary work spawn? That is another question. The answer is always: unbounded.

Re:Why literature is the ultimate big-data challenge by killjoywashere on Friday March 03, 2017 @06:27PM (#13787895)
> The answer is always: unbounded.

But after initial perturbations die out, the growth rate will likely look something like O(log n) or less.

Re:Why literature is the ultimate big-data challenge by baddox on Friday March 03, 2017 @12:43PM (#13784560)
Sounds like you may be interested in https://en.m.wikipedia.org/wiki/Kolmogorov_complexity
Re:Why literature is the ultimate big-data challenge by hyporthogon on Friday March 03, 2017 @02:50PM (#13785723)
Everything I've read by Chaitin is interesting and thoughtful, but this piece is particularly accessible and entertaining: https://arxiv.org/abs/math/0303352
Re:Why literature is the ultimate big-data challenge by killjoywashere on Friday March 03, 2017 @06:21PM (#13787856)
> 1TB ... is a lot of data

"You have a terabyte of data? For $30,000 you can put that in main memory. If your dataset fits in main memory, it's not big data." Source: Michael Stonebraker, January 2015.

Re:Why literature is the ultimate big-data challenge by woliveirajr on Friday March 03, 2017 @12:05PM (#13784243)