fedegr
We also want to show with this experiment the importance of creating benchmarks. In many use cases, practitioners choose more sophisticated models because they think this will give them better accuracy. The main idea is that robust benchmarks should always be created.
We noted that the paper only compares NeuralProphet against Prophet and does not include standard time series datasets (such as M-competitions). So we decided to test the model against simpler models (ETS in this case) using the StatsForecast library (https://github.com/Nixtla/statsforecast/).
Is linear regression better than prophet? Zillow benchmark
Prophet vs. Linear Regression on Real Estate: The Zillow Case
1. It is a fully automated end-to-end pipeline for forecast generation. The pipeline considers preprocessing such as missing value imputation, feature generation (static and dynamic), forecast generation, and also a module to validate forecasts on important time series competition data sets.
2. Users can deploy the pipeline in their cloud quickly. We use terraform (https://github.com/Nixtla/nixtla/tree/main/iac/terraform/aws), so it is very simple to deploy the pipeline on AWS. We are working to release versions of terraform on other clouds such as Azure and Google Cloud.
3. Users can use their own models. Just create a fork of the repo and make the appropriate modifications to include any model the user wants to deploy. On our side, we are working to include Deep Learning models with the nixtlats library (https://github.com/nixtla/nixtlats/) that we also developed.
About benchmarking using statistical models, we highly recommend using statsforecast (https://github.com/Nixtla/statsforecast) that we created. It is designed to be highly efficient in fitting statistical models on millions of time series. More complex models can be built on the results to get a positive Forecast Value Added.
We agree that in most cases prophet is not a good benchmark; however, we wanted to use it because it is one of the most used libraries in forecasting. For that reason, we also tested the solution against AWS Forecast obtaining better results.
Besides the better performance and scalability, the pipeline we created considering all the stages of time series forecasting: preprocessing (e.g. missing value imputation), creation of static and dynamic features, forecast generation, and finally evaluation using data sets of important competencies. (https://github.com/Nixtla/tsfeatures)
On the deployment side, the entire pipeline can be quickly deployed in the user's cloud using terraform. This allows for less development time. (https://github.com/Nixtla/nixtla)