News for hackers. Stuff that matters.

A comment on Bounter: A Python counter that uses limited memory regardless of data size. Parent comment

Re:Bounter: A Python counter that uses limited memory regard... by gizmo on Saturday October 21, 2017 @09:03AM (#15521719)
> The description should be more upfront that you are getting inaccurate answers.

You can't really expect a data structure using a finite amount of memory to contain an infinite amount of data. Otherwise it wouldn't be advertised as a "counter", but as a magic bag of holding.

[ Parent | on HN ]
Re:Bounter: A Python counter that uses limited memory regard... by tedmiston on Saturday October 21, 2017 @10:05AM (#15521875)
Containing an infinite amount of data and giving accurate answers are saying different things though. It can have finite memory and still give 100% accurate answers by batching from disk.

Chunking / iterative processing like this is an approach people commonly use with pandas which otherwise isn't great for data sets too big for RAM.

http://pandas-docs.github.io/pandas-docs-travis/io.html#iter...

[ Parent | on HN ]