Furthermore, often pre-DL methods can be recast as hand-tuned special cases of DL models - some sequence of linear operations with hand-picked discontinuities sprinkled around. If you can implement the pre-DL method using standard neural network components, then gradient descent training of a neural network "should" find an equivalent or better solution.
The M# competitions have usually shown very old forecasting algorithms work quite well, with frankly, way less training overhead and data. Ensemble models usually do best, but for a lot of use cases, DL is probably overkill versus ARIMA or triple exponential smoothing.
This is quite untrue. DL methods work well when there’s a lot of data in closed domains. It works well by learning from corpuses of text and media where it can make reasonable interpolations.
When you don’t have enough data and you don’t have a known foundational model that you can do zero shot from, DL doesn’t work better than simpler conventional methods.
There are many optimal, or known near optimal, direct design algorithms for systems that are well characterized. Not only do you get a solution with guaranteed qualities, but any analysis, including verification of reliability and accuracy, and valid operating regimes, of such solutions go way beyond what deep learning models provide.
Second, just assuming that a new algorithm is better than a previous algorithm, without testing, is completely irrational. An anathema to the entire purpose and benefits of science, math, and rational research in general.
If you are going to propose switching algorithms for a long known class of problems, you need to measure the new algorithm against the previous state of the art.
Otherwise people will drive into unnecessary deadends, deploy pointlessly bad designs, forget important knowledge that still has value, and worst of all, lose out on what the older algorithms can suggest for improving newer algorithms, with no excuse but carelessness.
There used to be a small hobbyist subculture on the Internet in the late 1990s that designed highly efficient approximate universal sequence predictor algorithms for the challenge of it. Now that AI is a thing, I've often wondered if there were some lost insights there on maximally efficient representations of learning systems on real computers. Most of those people would be deep into retirement by now.
Why don’t you share some citations?
I always enjoyed tracking down outre typewritten connectionist manuscripts from an author who had more time than compute.
It looks interesting, but I’m slightly confused about the way this is presented. It feels like it’s coming from the wrong angle.
Specifically, reducing a time series to a sequence of patterns and trying to predict what happens next is something that has been done for decades in some form or another. To me the unique aspect of this is that it fits the approach into a transformer.
So I’d expect to see comparisons against other approaches that do the same thing, not against other transformer approaches.
I wouldn’t be confused if the title was “Inverted Transformers are MORE EFFECTIVE THAN NORMAL TRANSFORMERS For Time Series Forecasting”.
However, if the target audience are transformer folk then it makes sense, it just seems that I’m looking at it from the other direction.