Re:Computer Scientists Invent an Efficient New Way to Count
by empath-nirvana on Friday May 17, 2024 @03:01PM
(#40393192)
If you don't know the length of the stream in advance, you can just calculate the margin of error when you're done, no?
Re:Computer Scientists Invent an Efficient New Way to Count
by cb321 on Friday May 17, 2024 @04:04PM
(#40393707)
You just solve for epsilon. That's what I did: https://news.ycombinator.com/user?id=cb321
Another practical thing is the "exception" if nothing is removed on line 6 in the original algorithm. This also seems needed for the proof but you would not want in production, though the chance of hitting it should be vanishingly small so maybe worth the gamble?
Here is my faithful interpretation of the algorithm. And then a re-interpretation with some "practical" improvements that almost certainly make the provability of the correctness impossible.
} }I tested it with Shakespeare's work. The actual unique word count is 71,595. With the second algorithm it is interesting to play with the threshold. Here are some examples.
threshold 1000 Mean Absolute Error: 2150.44 Root Mean Squared Error: 2758.33 Standard Deviation: 2732.61
threshold 2000 Mean Absolute Error: 1723.72 Root Mean Squared Error: 2212.74 Standard Deviation: 2199.39
threshold 10000 Mean Absolute Error: 442.76 Root Mean Squared Error: 556.74 Standard Deviation: 555.53
threshold 50000 Mean Absolute Error: 217.28 Root Mean Squared Error: 267.39 Standard Deviation: 262.84