News for hackers. Stuff that matters.
Re:Amazon's Exabyte-Scale Migration from Apache Spark to Ray... by parhamn on Wednesday July 31, 2024 @05:46PM (#41123806)
Im curious, how do data scientists use these massive datasets, especially the old stuff. Is it more of a compliance and need/should-save type thing or is the data actually useful? Im baffled by these numbers having never used a large BI tool, and am genuinely curious how the data is actually used operationally.

As a layman, I imagine lots of it loses relevancy very quickly, e.g Amazon sales data from 5 years ago is marginally useful to determining future trends and analyzing new consumer behavior regimes?

Re:Amazon's Exabyte-Scale Migration from Apache Spark to Ray... by gregw2 on Wednesday July 31, 2024 @05:50PM (#41123848)
If you have seasonal demand patterns, you generally need three years history to do good predictive analytics.

I do tend to agree data from five years ago is rarely relevant BUT our business is still using for some BI purposes data from the fiscal year before COVID as a comparison baseline for certain analytics/business processes which have been slow to reach pre-COVID levels of performance. So that means we are now using data 6 years old, comparing this year to that pre-COVID year for certain analytics!

Re:Amazon's Exabyte-Scale Migration from Apache Spark to Ray... by physicsguy on Thursday August 01, 2024 @01:33AM (#41126438)
Yeah 100%. I worked in wind energy for a while and the DS team would be pulling as much data as they could get to establish a baseline for normality due to seasonal trends in the wind. This also varied enormously around the world - for e.g. the UK is fairly windy all year, but India typically gets 2/3 of it's generated wind energy in the monsoon season which is about 3 months.
Re:Amazon's Exabyte-Scale Migration from Apache Spark to Ray... by smfjaw on Wednesday July 31, 2024 @06:45PM (#41124320)
I work in finance and it's great having big historical datasets, even if the figures are far lower in previous years it's good to see system 'shocks' and these can be used at a different magnitude/scaled for future forecasting
Re:Amazon's Exabyte-Scale Migration from Apache Spark to Ray... by refset on Thursday August 01, 2024 @08:22AM (#41128400)
Yep backtesting is critical step in the deployment of new trading algorithms.

Related, I rather enjoyed reading this other thread from June: "Ask HN: Is KDB a sane choice for a datalake in 2024?" https://news.ycombinator.com/item?id=40625800

Re:Amazon's Exabyte-Scale Migration from Apache Spark to Ray... by mr_toad on Thursday August 01, 2024 @01:09AM (#41126344)
Customer retention, infrequent purchases, and time series forecasts all benefit from having at least several years of data.
Re:Amazon's Exabyte-Scale Migration from Apache Spark to Ray... by jedberg on Thursday August 01, 2024 @01:51PM (#41131660)
Training a foundation model that understands shopping queries. 25 years of shopping data goes a long way.