News for hackers. Stuff that matters.

maxmc

Re:TimeGPT-1 on Friday October 13, 2023 @08:26PM (#37877027)
Hi,

Max from Nixtla here. We are surprised that this has gained so much attention and are excited about both the positive and critical responses. Some important clarifications:

The primary goal of this first version of the paper is to present TimeGPT-1 and showcase our preliminary findings from a large-scale experiment, demonstrating that transfer learning at this scale is indeed possible in time series. As mentioned in the paper, we deeply believe that pre-trained models can represent a very cost-effective solution (in terms of computational resources) for many applications. Please also consider that this is a pre-print version. We are working on releasing a reproducible set of experiments on a subset of the data, so stay tuned!

All previous work of Nixtla has been open source and we believe TimeGPT could be a viable commercial product, offering forecasting and anomaly detection out of the box for practitioners. Some interesting details were omitted because they represent a competitive advantage that we hope to leverage in order to grow the company and keep providing better solutions and continuing to build our ecosystem.

As some others have mentioned in the thread, we are working to onboard as many people as possible into a free trial so that more independent practitioners can validate the accuracy for their particular use cases. You can read some initial impressions of the creators Prophet [1] and GluonTS [2] or listen to an early test by the people from H20 [3]. We hope to see some more independent benchmarcks soon.

[1] https://x.com/seanjtaylor/status/1694745912776749296?s=20 [2] https://www.linkedin.com/posts/tim-januschowski_foundational... [3] https://youtu.be/N0gyDVUFPlg?si=xH8oy5cjgLm-o_WD&t=457

Re:TimeGPT-1 on Friday October 13, 2023 @07:47PM (#37876816)
We love ARIMAs. That is why we put so much effort into creating fast and scalable Arimas and AutoArima in Python [1].

Regarding your valid concern. There are several reasons for the high computational costs. First, ARIMA and other "statistical" methods are local, so they must train one different model for each time series. (ML and DL models are global, so you have 'one' model for all the series.) Second, the ARIMA model usually performs poorly for a diverse set of time series, like the one considered in our experiments. The AutoARIMA is a better option, but its training time is considerably longer, given the number and length of the series. Also, AutoARIMA tends to be very slow for long series. In short: for the 500k series we used for benchmarcking, ARIMA would have taken literally weeks and would have been very expensive. That is why we included many well-performing local "statistical" models, such as the Theta and CES. We used the implementations on our open-source ecosystem for all the baselines, including StatsForecast, MLForecast, and Neuralforecast. We will release a reproducible set of experiments on smaller subsets soon!

[1] https://nixtla.github.io/statsforecast/docs/models/arima.htm...

Re:Statistical methods outperform Amazon’s ML Forecast on Monday December 12, 2022 @12:16PM (#33957301)
Using 30,490 series of daily sales at Walmart (M5) we show that, Amazon Forecast is 60% less accurate and 669 times more expensive than running an open-source alternative in a simple cloud server.
Re:Statistical vs. Deep Learning forecasting methods on Thursday December 01, 2022 @12:53PM (#33819851)
A lot of M3 datasets we use are high-frequency, with large seasonal inputs. Considering Gaussian Processes (GP) complexity is O(N^3), a careful study of their performance would be challenging.

Also... I'm not aware of any efficient GP Python implementations.

Re:Statistical vs. Deep Learning forecasting methods on Thursday December 01, 2022 @12:41PM (#33819660)
Bonferroni's correction on hold-out data is an excellent suggestion. To adapt it into time series forecasting, one could perform temporal cross-validation with rolling windows and follow the performance's variance through time.

Unfortunately, the computational time would explode if the ML method's optimization is performed naively. Precise measurements of the statistical significance would crowd out researchers except for Big Tech.

Re:Statistical vs. Deep Learning forecasting methods on Thursday December 01, 2022 @12:31PM (#33819508)
Thanks for the comment!

In Machine Learning literature, the variance of accuracy measurements originates from different network parameters initialization. Since the deep learning ensembles already use aggregate computation in the hundreds of days, computing the variance would elevate the computational time into thousands of days.

In contrast, statistical methods that we report optimize convex objectives; their optimal parameters are deterministic.

That being said, we like the idea of including cross-validation with different splits for future experiments.

Re:Statistical vs. Deep Learning forecasting methods on Thursday December 01, 2022 @11:29AM (#33818532)
Comparison of several Deep Learning models and ensembles to classical statistical models for the 3,003 series of the M3 competition.
Imagine you want to forecast the next day of electricity load. We ran the test with PJM data.

Results: SeasonalNaive is 20% more accurate than NeuralProphet and 366 times faster. StatsForecast's MSTL is 24% more accurate than NeuralProphet and 11 times faster.

SeasonalNaive is 23% more accurate than Prophet and 75 times faster. StatsForecast's MSTL is 26% more accurate than Prophet and 2 times faster.

In the tutorial, we show how to Forecast 1M Time Series in 15 Minutes with Spark, Fugue, and Nixtla’s Statsforecast.
Figure 7 of the mentioned paper evaluates FB-Prophet in a extremely convenient environment of long horizon h in {30,60,90,120,150,180}. It is known that ETS and ARIMA models concatenate errors and degrade in performance with longer forecasting horizons. We have explored and offered solutions to these issues with the N-HiTS model specialized in long-horizon (https://arxiv.org/abs/2201.12886).

In recent years, FB-Prophet has gained a reputation for the poor quality of its predictions in many practical scenarios (short/medium term horizon) and its slow performance on bigger data sets; our ARIMA/ETS work and NeuralProphet confirmed those suspicions. The mentioned 92 percent improvements of the paper are restricted to h in {1,3,15,60} (https://arxiv.org/pdf/2111.15397.pdf).

This post's results are rather for short-horizon tasks, the same as NeuralProphet experiments. But we are confident that specialized tools like N-HiTS would outperform Prophet in long-horizon settings.

This comparison was not intended as an academic paper. However, we are confident that predicting and evaluating performance over more than 55k series from the M international forecasting competitions (standard benchmarks in the field) is a good start. After corresponding with the author, we included additional data sets from the electricity domain like Ercot and ETTM2. The dataset that did not finish training shows that Neuralprophet simply does not scale. (We let the training run for over 73 hours with 96 CPUs -as NeuralProphet implementation is restricted to CPU-, and we canceled it after investing 288 USD in it) Suggestions to strengthen our experiments (without spending hundreds of dollars) are highly welcomed.

Regarding the hyperparameter selection, we went beyond the original NeuralProphet paper, tunning and actively trying to help improve its performance. We even made a PR fixing NP bugs in the process.

Be surprised. Here is the original Neuralprophet paper: https://arxiv.org/pdf/2111.15397.pdf It only compares itself to Prophet.

Exponential Smoothing: faster and more accurate than NeuralProphet

https://github.com/Nixtla/statsforecast/blob/main/experiments/neuralprophet/README.md

We benchmarked on more than 55K series and show that ETS improves MAPE and sMAPE forecast accuracy by 32% and 19%, respectively, with 104x less computational time over NeuralProphet.We hope this exercise helps the forecast community avoid adopting yet another overpromising and unproven forecasting method.

( Read More... | 42 comments | 161 points | on HN )
Re:Exponential Smoothing (ETS) for Python on Tuesday July 19, 2022 @03:16PM (#32156569)
Optimized using numba. 400% faster than statsmodels and 1.5x faster than R, with improved accuracy and robustness.