I think this is a reasonable point, but I would just add that a lot of people in CS academia are well aware of this. The problem is that we all serve multiple masters and one of the things we have to do is publish frequently. I think you'll find that many CS academics try to strike a balance between publishing for the sake of publishing and actually working towards a larger scientific goal. Personally speaking I'm definitely guilty of publishing work that looks good on my CV, but does not advance my deeper scientific agenda. That said the science is always on the front of my mind even if it only makes up 10-20% of my actual publications.

In regards to solutions it would be great if we focused less on the frequency at which we published and editors were more willing to publish work that had novel ideas even if it did not have state of the art performance (yet). Although like any job there will always be parts that are tedious, involve politics and yes parts that are even counter productive. At some point as an individual you just have to play the game while still thinking about and trying to advance the bigger picture.

>Personally speaking I'm definitely guilty of publishing work that looks good on my CV, but does not advance my deeper scientific agenda.

This sounds analogous to resume driven development. And I completely understand it. I have pragmatically chosen the best tech for the job (factoring in the learning cure for new tech and what I already know) and haven't learned much new tech for the last couple of years. Now I get the feeling my CV is looking a bit dated. Does it make me a worse at developing software? I would say not. I have mastery in a few areas rather than shallow knowledge in a lot. Will it make it harder for me to get a job? Quite possibly if I keep going this way.

This sort of comment does very little to advance the conversation. I made several attempts to write a response that talks about heuristics, algorithms, what industry does and what academia does, but in the end I found your portrayal of academia and industry to be so hyperbolic and unrealistically one sided that I felt it not even worth getting into the discussion.

Let's try to have respect for each other on both sides of the "academia/industry" fence and not resort to unrealistic black and white views of the world.

If you look at an article talking about finances and machine learning, the success depends if you earn real money or not. It is cynical but it is the way algorithms are valued in that space. Real money returns are the real test.

> Most academic CS literature is complete BS

Thats complete BS. Certain journals are complete BS, and certain subfields of CS are superfluous and often times lack substance in publications. But CS literature, from top tier journals, has vast merits.

English is not my native language so I don't get offended at all by the coarse language (coming from "saltier" spanish, for me it's actually quaint that you just hint at the word with 'BS'), but please. I'm sure that with a little more effort you both can provide an adjective that provides more information. Just tagging things as BS seems juvenile, and in HN we try to keep conversations more rational and less emotional.

Depends. If the the method it proposes do have an advantage over other methods, and generalizes well, it is a good paper nevertheless.

I think the assertion is that they never do. This is widely regarded in the ml classes as the advantage of data being much more important than the algos.

I'd be curious to know how accurate the assertion is.

Most of them are. But my problem with this claim is, good papers could still be classified into this formula as well. If talking about field like image recognition, almost everything after ResNet is incremental improvement to it, most of them are BS, but there are still some interesting new ideas even their gain is just marginal.

I think the difference is that many of the new algos aren't even showing reproducible incremental improvements. Rather, they just benefit from the massive amounts of data available.

Though, I fully grant that the problem is as much that they are optimizing different things. Sometimes, the goal was just faster training, not better accuracy. (Though, I'm curious how fungible those are... Something that trains faster would ostensibly train for better accuracy at the same times, no?)

Could an enterprising person find such papers, cleverly select a dataset such that X is way worse, and finally refute the original paper?

This seems like something an amateur could do. Wouldn't that tear down a bunch of this junk?

Sure. But for approximately the same level of effort, you could generate a new paper in the same template. It would likely be easier to get that "novel result" published than the refutation.

And that's generally the truth in every scientific field.

Right now you have a "reproduction renaissance" in psychology and cancer, where you can get published, and get acknowledged, if you tear down a high level paper. But in general, unless you aim at refuting the highest level, it is hard to get published and advances your career nowhere, so people have no incentive to do it.

A non-replication does not "tear down" a paper... It is not any kind of "attack", your attitude is one of the sources of the problem.

Edit:

When I did medical research I wished people would try to replicate my findings so I could see how consistent the results were and how well my model fit future data... In fact it was really annoying that it was unlikely anyone would ever try to do a direct replication.

My attitude? What attitude is that?

A single non-replication does not. Consistent failure to replicate, after communicating with the original author, does.

See e.g.

http://www.slate.com/articles/health_and_science/cover_story...

http://www.nature.com/nature/journal/v483/n7391/full/483531a...

Or just google "reproducibility crisis", the two most common subjects that come up are cancer research and psychology research.

That is the problem: Why wouldn't try to do a replication?

Because the incentives are to provide "new research", not to confirm or deny a known result.

It seem to me that most probably no-one has gotten tenure by reproducing, disproving results.