News for hackers. Stuff that matters.
Re:Global Forecast System source code by davidwihl on Saturday January 31, 2015 @10:30PM (#8979060)
Independently I was just wondering tonight why weather forecasting does not use some Machine Learning techniques as there is an abundance of training data. Wouldn't any number of regression ML algorithms find a more accurate prediction model? If Google did weather....
Re:Global Forecast System source code by desdiv on Saturday January 31, 2015 @11:39PM (#8979187)
ML techniques certainly are being used to predict the weather, maybe just not at NOAA or the Weather Channel. The weather derivative industry[0] is an $8 billion dollar industry, making it larger than the entire rocket launch industry (public and private combined). Just as there are Elon Musks in the rocket launch industry, there are maverick hedge funds out there making a killing by predicting the weather using the latest Big Data tools.

[0] http://en.wikipedia.org/wiki/Weather_derivative

Re:Global Forecast System source code by darksaints on Saturday January 31, 2015 @10:48PM (#8979099)
Sometimes Machine Learning is the wrong approach. Direct mathematical modeling, if the underlying causal phenomenon are well known, bypasses the whole learning aspect of an explanatory system. Machine learning might help make sense of the residual noise that can't be accounted for with direct mathematical modeling. But at the level of sophistication that I've seen (predicting wind patterns using Finite Element Analysis methods mapped to topographical patterns fine-grained enough to model small buildings), I can't imagine any machine learning methods that could make any sense of the residual noise.
Re:Global Forecast System source code by davidwihl on Saturday January 31, 2015 @11:31PM (#8979170)
I agree that Machine Learning is not always the right approach but wouldn't it be a relatively inexpensive experiment to see if it could be used for more accurate predictions? I'm surprised that topographical patterns are mapped that precisely when ASOS and AWOs stations are so far apart. Also the weather data from commercial airplanes is so far above the surface that modeling small buildings would be meaningless.

I heard a talk from ex-Googler climate.com (now Monsanto) guys. Their forecasts were accurate enough to build a very profitable insurance business. I bet they used Bayesian rather than FEM techniques.

Re:Global Forecast System source code by bainsfather on Sunday February 01, 2015 @05:10AM (#8979596)
Is it that 'physics simulation' forecasting is best for short timescales (e.g. < 1 week in the uk), whilst ML can be better for estimating next year's weather statistics in some particular place? (Because your butterfly effect means that simulating the actual weather out to 1 year 'accurately' is not possible).

Presumably any short-term ML forecasting would use the physics simulation as it's main input, and then try to improve slightly on it (e.g. maybe you observe that in a particular small area, the rainfall is on average 10% more than the physics simulation, so your ML method would add 10%, giving you a slight forecast improvement)?

Re:Global Forecast System source code by GFK_of_xmaspast on Sunday February 01, 2015 @08:22AM (#8979814)
Probably because if it worked they'd already be doing it.