Re:A decoder-only foundation model for time-series forecasting
by remontoire on Friday February 02, 2024 @10:50PM
(#39237388)
Probably drawing an analogy to how causal pretrained models go through stages of understanding language, words -> grammar -> meaning. Gwen mentions this experience when training character level RNNs. https://gwern.net/scaling-hypothesis#why-does-pretraining-wo...
Re:A decoder-only foundation model for time-series forecasting
by horacemorace on Friday February 02, 2024 @11:09PM
(#39237471)
Totally. Also basic temporal scale or cyclic properties. It’s kind of mind blowing that the shape of most recorded human patterns is reducible in this way.
Re:A decoder-only foundation model for time-series forecasting
by nerdponx on Friday February 02, 2024 @10:55PM
(#39237408)
It's a bit of an anthropomorphism ation. I don't believe it has any formal meaning here. The idea is that there are certain kinds of underlying signals and patterns which are common to a wide range of time series data. So if a model is able to learn those signals and patterns, it can look at any time series and, with enough historical data, predict future data, without actually updating any model weights. Those signals constitute a "grammar" of sorts.
Re:A decoder-only foundation model for time-series forecasting
by WanderPanda on Friday February 02, 2024 @10:31PM
(#39237300)
They probably mean the manifold of the universe
Re:A decoder-only foundation model for time-series forecasting
by mnoronha on Saturday February 03, 2024 @02:01AM
(#39238156)
ah yes, the manifold of the universe (god) :)
Re:A decoder-only foundation model for time-series forecasting
by alextheparrot on Friday February 02, 2024 @10:46PM
(#39237367)
You could probably consider learning a sign wave to be a “grammar” related to periodic variations. Grammar in this context feels like “What are the core conceptual heuristics that help guide towards faster understanding”
Can someone elaborate on what a grammar means in the context of time series forecasting?