Re:Stan is a state-of-the-art platform for statistical modeling
by elsherbini on Wednesday December 23, 2020 @10:16AM
(#25518390)
Does anyone know what a practical upper limit is for Stan in terms of size of data set / number of parameters to fit? Can you use stan to fit a model with ~10^4 parameters and ~10^6 rows of data if you had access to ~10^3 cores? How long would it take?
[ Parent
| on HN ]
Re:Stan is a state-of-the-art platform for statistical modeling
by standevbob on Wednesday December 23, 2020 @10:44AM
(#25518725)
Yes, we regularly use Stan's MCMC to fit relatively simple time-series regression models or item-response theory type models with 10^5 parameters and 10^6 rows of data on a desktop computer. It can take a day, though. It's much faster with variational inference, but that can be less stable and it doesn't give you the same uncertainty quantification because of the way the KL-divergence is ordered in the objective.
Stan can parallelize multiple chains and it can parallelize the density/gradient calculations in a single chain. But for the latter to be efficient, the chunks being parallelized need to be compute intensive, like you might get in a pharmacometric compartment model where you might have to solve a bunch of differential equations for each of thousands of patients in a clinical trial.
Re:Stan is a state-of-the-art platform for statistical modeling
by hendzen on Wednesday December 23, 2020 @10:31AM
(#25518567)
Stan used to only be able to parallelize across chains but they introduced within-chain parallelism this year. Even then some of the work is still serial so I don’t think you can expect linear speedups past a certain point.
Re:Stan is a state-of-the-art platform for statistical modeling
by gbrown on Wednesday December 23, 2020 @10:27AM
(#25518521)
I imagine it depends on which algorithm you’re using - maybe with their VB functionality (I almost entirely use the souped up NUTS algorithm for full Bayesian inference).
It also likely depends on how well conditioned your model is - even if you can get it to run for huge models on reasonable hardware, convergence may not be practical.