Re:A Petabyte of Health Insurance Prices Per Month – Why is ...
by ramraj07 on Tuesday July 11, 2023 @03:46PM
(#36686325)
I work with petabytes of data. A petabyte of text is an insane amount of text. You need a billion X multiplication of basic data. That’s not just combinatrionics, it’s deliberate obfuscation.
[ Parent
| on HN ]
Re:A Petabyte of Health Insurance Prices Per Month – Why is ...
by abraae on Tuesday July 11, 2023 @03:54PM
(#36686442)
4 tables, each with 1k rows and one of them with a 1K text column in it, joined together to give a cartesion product, will give you a petabyte of data. So it doesn't sound impossible (just absurd).
Re:A Petabyte of Health Insurance Prices Per Month – Why is ...
by grumple on Tuesday July 11, 2023 @07:00PM
(#36688366)
That's a very artificially bloated way of compounding the data in memory with lots of data duplication. From storing the example tables you used, you'd need to store 62.5 million times that to get a petabyte of data. It's an absurd amount of data.
Re:A Petabyte of Health Insurance Prices Per Month – Why is ...
by ramraj07 on Wednesday July 12, 2023 @12:15AM
(#36690404)
even taking your contrived example, you have to store it in the stupidest way possible to get that level of storage - even assuming your 1K of text is incompressible, you only have 1K unique values. The most basic compression algorithms will make easy work out of compressing the crap out of it, leave alone the advanced compression algorithms used for any big data storage formats.