What? Who said “math gives nothing”? I spend most of my day building things out of math. I think math and scientific inquiry are basically the most important tools invented/popularized in the past 1000 years.

The lapse here is not math, but rather spending lots of attention on abstract thought disconnected from any kind of reality check. Of course, there’s nothing inherently wrong with speculating sans evidence about the future, it’s generally a harmless hobby. In the best case it makes for fun SF novels. Convincing yourself, still without direct evidence, that your speculation reflects truth implies that something has gone off the rails in the reasoning process, however.

Using statistical analysis to understand real causal relationships in areas we have real data about is damn hard, and even plenty of people who are highly trained as statisticians screw up all the time. Academic fields like comparative politics (to take an example I spent a fair amount of time studying) are rife with poor conclusions drawn from bad analysis. The LessWrong folks are hardly unique in applying logic poorly. But they do tend to tackle more speculative questions and convince themselves more firmly of their conclusions (at least, such is my impression as an outsider).

I think this is a common problem when people working in fields that have somewhat accurate mathematical models look at fields that don't. They often don't realize how hard it is to create an accurate mathematical model for many situations, and assume that the other fields don't have them because the individuals who work in said fields aren't as good at math.

Which is why every so often you'll get things like a physicist spending a couple months studying economics in their free time and deciding that they can now unlock the secret to economics which has eluded economists.

This xkcd comic sums it up well:

https://xkcd.com/793/

The more you extrapolate the more frequently you will need to adjust your future predictions because of errors in your initial measurements. This goes for any kind of extrapolation (for instance: plotting a course on a map), but it goes even more for extrapolating the future from limited evidence present today. Your 'best guess' might be off by many orders of magnitude if the evidence you have today is only loosely related to the future in terms of importance and where evidence may not be nearly as independent in nature as you currently perceive.

This can lead to your best guess based on available evidence being about as good at predicting the future as randomness in spite of all the apparent effort at making the predictions mathematically sound.