hrzn

A few years back I made a modest attempt at writing a concise yet readable sudoku solver in Python - in about 29 lines: https://github.com/hrzn/sudoku/blob/master/sudoku.py

Could have been made shorter at the price of readability.

We likely overestimate AI's short-term impact, and there might even be a financial bubble about to pop. But I also think we underestimate the long-term impact. We're building absolutely amazing capabilities faster than many would have thought possible only a few years ago - I especially think applications to science and engineering will be huge and transformative.

> Python had nothing comparable

This is in part why we built Darts. Now I think we can say the situation is quite different. Darts offers many things offered by the R forecast package, and then some (for instance the ability to train ML models on large datasets made of multiple potentially high-dimensional series).

I would recommend Darts in Python [1]. It's easy to use (think fit()/predict()) and includes

* Statistical models (ETS, (V)ARIMA(X), etc)

* ML models (sklearn models, LGBM, etc)

* Many recent deep learning models (N-BEATS, TFT, etc)

* Seamlessly works on multi-dimensional series

* Models can be trained on multiple series

* Many models offer rich support for probabilistic forecasts

* Model evaluation is easy: Darts has many metrics, offers backtest etc

* Deep learning scales to large datasets, using GPUs, TPUs, etc

* There's even now an explainability module for some of the models - showing you what matters for computing the forecasts

* (coming soon): an anomaly detection module :)

* (also, it even include FB Prophet if you really want to use it)

Warning: I'm probably biased because I'm Darts creator.

[1] https://github.com/unit8co/darts

Note that the actual EV of a Cauchy random variable is undefined...

You can install darts with conda using

    conda install u8darts-all

We are working on improving our support of conda based installs.

You can try installing it using

    pip install 'u8darts[torch]'

This will give you everything besides Prophet and pmdarima.

I would say that compared to Greykite, Darts really attempts to unify a wide variety of forecasting models under a common simple and user-friendly API. There are many differences, but for instance, AFAIK there's no deep learning model in Greykite (it focuses on two algorithms: their built-in algorithm and Prophet), whereas Darts tries to lower the barrier for using deep learning models for forecasting. Crucially for ML-based models, it also means being able to train on multiple (possibly thousands or more) of possibly multi-dimensional time series.

In short: yes, at least anomaly detection. That's one of the main things on our to-do list.

Oh, nice, I didn't know that. I've edited my reply, thanks!

We do have ensemble models. Something like

    model = NaiveEnsembleModel([model1, model2, ...])
    model.fit(my_series)
    prediction = model.predict()

Will return an average prediction. Look at RegressionEnsembleModel for an ensemble model which uses a regression model to learn how to combine the individual forecasts.

At the moment Darts doesn't have hierarchical reconciliation methods (if that's what you meant), but it's on the backlog :)

Thanks for the feedback, I absolutely agree about the need for easy-to-use tools for dealing with time series. This is exactly the motivation that prompted us to work on Darts initially.

I like your suggestions of adding comparison to the few other libraries out there, as well as explaining the need for having our own TimeSeries data structure. We should try to do that sometime soon.

Concerning dependencies, we already have some dependencies as extras. "pip install darts" will install everything, but "pip install u8darts" will install only the core (without Prophet and pmdarima), or "pip install u8darts[torch]" only the core+pytorch models.

> Or is it just turning every column in my pandas dataframe into a series to pass into the covariates array?

Basically if you have a multivariate series represented as a pandas dataframe with several columns, the way to go is to create your TimeSeries by calling TimeSeries.from_dataframe(my_df). That will return a multivariate time series.

We don't yet have a discord channel, but I'm planning to open a Slack channel sometime soon. If you have other questions feel free to drop me an email: [email protected]

You're welcome, glad you like it and thanks for the feedback :)

In some cases Darts is wrapping around existing models (like Prophet, or statsmodels-based models for instance); in other cases we wrote our own implementations, so it's really a mix.

Hi! I'm one core developer (and creator) of the library. Thanks for all the comments. I just wanted to highlight a couple of things that we think are quite cool about Darts:

* It makes using all sorts of forecasting models (from ARIMA to deep learning) easy, using fit() and predict(), similar to scikit-learn.

* It's easy to fit deep learning and other ML-based models on multiple time series, potentially on big datasets too. The time series can be multivariate.

* Darts is not only wrapping existing models. We also have our own implementations, for instance of TCN (Temporal Convolutional Networks), or adaptations N-BEATS (which we extended to handle multivariate series), DeepAR and others.

* Darts makes it very easy to include past and/or future covariates as inputs for the predictions.

* Some models offer probabilistic forecasts; sometimes with the possibility to configure your favourite likelihood function (e.g. Gaussian for continuous values or Poisson for discrete values).

* Everything uses the "TimeSeries" class, which makes the API consistent across tools and models, and make it harder to make mistakes. For instance it's easy to consume the output of one model by another model, and all models can be backtested the same way.

This is supported but only by neural-nets models, which are fit using SGD, hence naturally not requiring the whole dataset in memory. Other models like ARIMA do need the full series loaded in memory.

The models that work on multiple time series in Darts accept Sequence[TimeSeries] for their fit() method. These sequences can either be Lists (fully in memory, simplest option), or when needed it can be a custom Sequence which for example does lazy loading from disk (somewhat similar to what PyTorch Datasets are doing) with the __getitem__() method.

If you need even more control, for instance because you have only one very long series that doesn't fit in memory then you can implement your own Darts "TrainingDataset". In this case you can control how to slice your series exactly.

Edit: I realised this only answers the first sentence of your comment ;) For now there's no mechanism for scaling to multiple machines beyond what PyTorch is already offering. AFAIK it's reasonably easy to scale to multiple GPUs on a machine, but I'm not sure how it would scale on several machines. We never had to try this yet! (Note that actually a single CPU can handle training deep nets models on 10's of thousands of time series similar to the M4 competition in a fairly reasonable time).

Both - some models are wrapped (like ARIMA & ETS around statsmodels, Prophet around fbprophet) and we write others ourselves (RNNs, TCNs, N-Beats, ...). Basically we take a pragmatic approach here, we do whatever is best to use a given model in Darts.

For training forecasting models on multiple time series (and potentially large datasets), you can take a look at Darts [1] and the blog post [2].

[1]: https://github.com/unit8co/darts/

[2]: https://medium.com/unit8-machine-learning-publication/traini...

For those interested in time series library, we are developing Darts [1], which focuses on making it easy & straightforward to build and use forecasting models. Out of the box it contains traditional models (such as ARIMA) as well as recent deep learning ones (like N-Beats). It also allows to easily train models on multiple time series (potentially scaling to large datasets), as well as on multivariate series (i.e., series made of multiple dimensions). It will soon support probabilistic forecasts as well.

[1] https://github.com/unit8co/darts/

A lot of these issues also have to do with the publishing system.

There is no incentive in publishing stuff that don't work even though they were reasonable things to try. This indirectly pushes a lot of bad/flawed/incomplete papers and results to be published. Even if your methodology is flawed but you try hard enough, you have a >0 probability of getting your paper published somewhere.

As part of a solution, I think we would benefit from the existence of prestigious journals for negative results, which would incentivise also publishing what doesn't work. This way researchers wouldn't have to try massaging their data and experiments until it looks like it works. They could just publish that it doesn't work, and it would be good for them too.

Even if these "noticeable" sequences make up only a tiny fraction of all possible sequences, the probability that they show up at least once in a while among all of the world lotteries is actually high.

In general the probability that some unlikely events occur is high.