While I completely agree with you on the massive backlog of problems that could be solved with very simple data analysis (this is the reason I am in the field), your thoughts about data modeling techniques are very much off the mark and sound like something read from a dated statistics textbook.
First, on logistic regression:
-I can tell you with certainty that logistic regression is not vastly underused, and is consistently among the most popular machine learning algorithms [1]. From personal experience I think it's likely the most powerful model there is.
-People are excited about Bayesian models and NN because they work. Actually, most logistic regression in practice is Bayesian by nature (regularization much?). More so, logistic regression is actually a piece of what powers NNs, and it's incorrect to directly compare the two. To put it more blunt, these are drastic improvements to logistic regression that allow the technique to solve greater problems.
-You are trivializing the amount of training needed to properly apply a classifier by calling it simple.
Second, there are plenty of basic questions in the data field to answer [2]:
-Doing anything with large amounts of data is still complicated and hard
-Unifying multiple data sets
-General model comparison
-Most modeling assumptions and as a result conclusions reached from them are wrong
-Communicating results to people. This is the most underestimated. One of the reasons we have such a backlog of problems that haven't been touched by simple data analysis techniques is the difficulty involved in explaining results. P-values are fucking meaningless and massively misinterpreted and misused. I won't start on frequentist statistics.
1) http://www.quora.com/What-are-the-top-10-data-mining-or-mach...
And the computer scientist who eats Debbie's waffles every Saturday spends his weekends reading about SpaceX instead of pulling her data into R and giving her some insights that would give her a much safer retirement.
Back to space. There are some long shot wins with research like this, things that are pure research prerequisites for big changes. As in, once we've looked at and followed a comet, we'll have the experience doing remote landings on small rocks in space. Why not do that to asteroids? Perhaps we can find an asteroid with an unusually high concentration of rare earth metals. That could bootstrap fuel cell research by dropping the price of rare, useful materials like platinum. Or gold. Such tech would help alleviate many problems at once. It might break economies, but it may also put us post-scarcity for manufacturing tech.
I don't mean to make it sound like I don't think it's not worth doing just for its own sake. I really do. But I also think that the potential wins we're setting up for the next generation or three down the road make starting now worth it. Being slow and thorough is a risk of lost opportunity, potentially huge loss. I also think there's plenty of brilliance available to work on the other problems, too, and I am often confused that we feel a scarcity in that respect (like how there are food shortages despite incredible productive ability - I just can't properly wrap my head around the subtle complexities that cause such strange things).
So I think you have a point, but I think it's a bit harsh to call it nihilistic. That's certainly there, but there's more good to be had, too. Can we do better? Sure, but I'm cynical enough that I'll accept the occasional wins for what they are, while secretly hoping it changes to world for the better.
>Massive environmental destruction, humanitarian disasters.
Right?
How many time do you think you have?
I disagree. A downvote is a signal, "fewer comments like this one please". Sometimes it's obvious what that means, but sometimes it's not. In that case it's perfectly reasonable to ask, what was it about my comment that you disliked? A feedback signal is only useful if understood, and sometimes to be understood it needs clarifying.
> Make you wonder if our industry's mind share being wasted on web apps and consumer gadgets because of the profit incentive associated?
I find this sentiment strange. To me, your kind of excitement, which seems to be pure joy over incredible engineering feats, seems nihilistic. There are serious terrestrial problems happening right now. Massive environmental destruction, humanitarian disasters. Even in relatively stable places like the Bay Area I know many people who are teetering on personal disaster. These are extremely complex problems and engineers have extraordinarily powerful skills to help solve them.
I understand why people are captivated by things like comets and why they'd want to attempt these extraordinary feats. It's exciting and fun and there are few joys that can match doing something that's never been done before.
For that reason I don't begrudge anyone who chooses that kind of work. Earth's crises are incredibly depressing and there's largely no glory in helping a few people stay out of homelessness, or helping a kid get through college. It's risky, demoralizing work.
But I don't really understand why a mission like this would be considered a noble pursuit. Fun, sure. But not especially noble. Yes, basic space science has some value to society in the long term. But we have a MASSIVE backlog of basic science that is being unused. We have journals chock full of solutions to problems that are going unimplemented.
Even something as simple as logistic regression, which has been around for 70 years, is vastly underused. People get excited about bayesian models and neural networks and all this great research, but there's still a massive backlog of problems that could be solved with a simple regression or an ANOVA. There are no basic questions to solve, just the work of obtaining and cleaning data, doing the analysis, and packaging it in a tool that people can actually use.
You seem to wonder why more people don't take on extremely difficult challenges on the edge of possibility. But I wonder why more people don't use existing technical tools to solve the massive backlog of problems that are being ignored.