Re:Amazon's Exabyte-Scale Migration from Apache Spark to Ray...
by gregw2 on Wednesday July 31, 2024 @05:50PM
(#41123848)
If you have seasonal demand patterns, you generally need three years history to do good predictive analytics.
I do tend to agree data from five years ago is rarely relevant BUT our business is still using for some BI purposes data from the fiscal year before COVID as a comparison baseline for certain analytics/business processes which have been slow to reach pre-COVID levels of performance. So that means we are now using data 6 years old, comparing this year to that pre-COVID year for certain analytics!
Re:Amazon's Exabyte-Scale Migration from Apache Spark to Ray...
by physicsguy on Thursday August 01, 2024 @01:33AM
(#41126438)
Yeah 100%. I worked in wind energy for a while and the DS team would be pulling as much data as they could get to establish a baseline for normality due to seasonal trends in the wind. This also varied enormously around the world - for e.g. the UK is fairly windy all year, but India typically gets 2/3 of it's generated wind energy in the monsoon season which is about 3 months.
Re:Amazon's Exabyte-Scale Migration from Apache Spark to Ray...
by smfjaw on Wednesday July 31, 2024 @06:45PM
(#41124320)
I work in finance and it's great having big historical datasets, even if the figures are far lower in previous years it's good to see system 'shocks' and these can be used at a different magnitude/scaled for future forecasting
Re:Amazon's Exabyte-Scale Migration from Apache Spark to Ray...
by refset on Thursday August 01, 2024 @08:22AM
(#41128400)
Yep backtesting is critical step in the deployment of new trading algorithms.
Related, I rather enjoyed reading this other thread from June: "Ask HN: Is KDB a sane choice for a datalake in 2024?" https://news.ycombinator.com/item?id=40625800
As a layman, I imagine lots of it loses relevancy very quickly, e.g Amazon sales data from 5 years ago is marginally useful to determining future trends and analyzing new consumer behavior regimes?