News for hackers. Stuff that matters.
Re:The algorithms that make Instacart roll by jfengel on Friday March 26, 2021 @10:02AM (#26592245)
Where are they getting "petabytes" of daily data from? Even if you took every single transaction anywhere near the industry, in all its details, and represented it in a verbose format like JSON or XML, that's still only kilobytes of data times billions of transactions = terabytes.

If they were gathering tons of video, maybe. Or vast numbers of high-precision sensors generating data continuously. But I didn't see anything like that.

It's just hard for any human activity to generate that much data. Is "petabytes" just some kind of hype, or is there something I'm missing?

Re:The algorithms that make Instacart roll by YoungWeb on Friday March 26, 2021 @10:10AM (#26592337)
To me it seems to be more of a volume data point rather than storage. So, it might only be terabytes of data, but for instance in ML algorithms, you could be running that data through the algorithm many times every two minutes. So by the end of the day, you've processed "petabytes" of data. This is just my interpretation of it. I could be wrong to.
Re:The algorithms that make Instacart roll by sn_master on Friday March 26, 2021 @02:57PM (#26595602)
A lot of big data datasets are de-normalized to make queries faster. Joining is very expensive.

Source: Worked in big datasets at Bing for a couple years.

Re:The algorithms that make Instacart roll by hnpobdo on Friday March 26, 2021 @03:34PM (#26596016)
1 user searches for a product, and then clicks on 1: although you estimate this transaction generates 1kb of data, in fact this is more on the order of 1kb x the number of candidates, so a few MB.
Re:The algorithms that make Instacart roll by TrainedMonkey on Friday March 26, 2021 @02:42PM (#26595426)
Maybe data streams collected from phones of people ordering groceries and buyers going to stores could add up to petabytes.