Re:The algorithms that make Instacart roll
by YoungWeb on Friday March 26, 2021 @10:10AM
(#26592337)
To me it seems to be more of a volume data point rather than storage. So, it might only be terabytes of data, but for instance in ML algorithms, you could be running that data through the algorithm many times every two minutes. So by the end of the day, you've processed "petabytes" of data. This is just my interpretation of it. I could be wrong to.
Re:The algorithms that make Instacart roll
by sn_master on Friday March 26, 2021 @02:57PM
(#26595602)
A lot of big data datasets are de-normalized to make queries faster. Joining is very expensive.
Source: Worked in big datasets at Bing for a couple years.
1 user searches for a product, and then clicks on 1: although you estimate this transaction generates 1kb of data, in fact this is more on the order of 1kb x the number of candidates, so a few MB.
Re:The algorithms that make Instacart roll
by TrainedMonkey on Friday March 26, 2021 @02:42PM
(#26595426)
Maybe data streams collected from phones of people ordering groceries and buyers going to stores could add up to petabytes.
If they were gathering tons of video, maybe. Or vast numbers of high-precision sensors generating data continuously. But I didn't see anything like that.
It's just hard for any human activity to generate that much data. Is "petabytes" just some kind of hype, or is there something I'm missing?