News for hackers. Stuff that matters.
Re:A decoder-only foundation model for time-series forecasting by neodypsis on Friday February 02, 2024 @10:31PM (#39237297)
> "Synthetic data helps with the basics. Meaningful synthetic time-series data can be generated using statistical models or physical simulations. These basic temporal patterns can teach the model the grammar of time series forecasting."

Can someone elaborate on what a grammar means in the context of time series forecasting?

Re:A decoder-only foundation model for time-series forecasting by remontoire on Friday February 02, 2024 @10:50PM (#39237388)
Probably drawing an analogy to how causal pretrained models go through stages of understanding language, words -> grammar -> meaning. Gwen mentions this experience when training character level RNNs. https://gwern.net/scaling-hypothesis#why-does-pretraining-wo...
Re:A decoder-only foundation model for time-series forecasting by horacemorace on Friday February 02, 2024 @11:09PM (#39237471)
Totally. Also basic temporal scale or cyclic properties. It’s kind of mind blowing that the shape of most recorded human patterns is reducible in this way.
Re:A decoder-only foundation model for time-series forecasting by nerdponx on Friday February 02, 2024 @10:55PM (#39237408)
It's a bit of an anthropomorphism ation. I don't believe it has any formal meaning here. The idea is that there are certain kinds of underlying signals and patterns which are common to a wide range of time series data. So if a model is able to learn those signals and patterns, it can look at any time series and, with enough historical data, predict future data, without actually updating any model weights. Those signals constitute a "grammar" of sorts.
Re:A decoder-only foundation model for time-series forecasting by WanderPanda on Friday February 02, 2024 @10:31PM (#39237300)
They probably mean the manifold of the universe
Re:A decoder-only foundation model for time-series forecasting by mnoronha on Saturday February 03, 2024 @02:01AM (#39238156)
ah yes, the manifold of the universe (god) :)
Re:A decoder-only foundation model for time-series forecasting by alextheparrot on Friday February 02, 2024 @10:46PM (#39237367)
You could probably consider learning a sign wave to be a “grammar” related to periodic variations. Grammar in this context feels like “What are the core conceptual heuristics that help guide towards faster understanding”