News for hackers. Stuff that matters.
Re:Failures of Deep Learning by Houshalter on Friday March 24, 2017 @09:19PM (#13953530)
Just yesterday there was a big discussion on here about how academic papers needlessly complicate simple ideas. Mainly by replacing nice explanations with impenetrable math notation in an attempt to seem more formal. This paper is very guilty of this.

E.g. page 5. They attempt to explain a really simple idea, that they generated images of random lines at a random angle. Then labelled the lines positive or negative examples, based on whether the angle was greater than 90 degrees or not. Then they take sets of these examples. And label them based on whether they contain an even or odd number of positive examples.

They take several paragraphs over half a page to explain this. Filled with dense mathematical notation. If you don't know what symbols like U, :, ->, or ~ mean, you are screwed because that's not googleable. It takes way longer to parse than it should. Especially since I just wanted to quickly skim the ideas, not painfully reverse engineer them.

Hell, even the concept of even or odd, is pointlessly redefined and complicated as multiplying + or - 1's together. I was scratching my head for a few minutes just trying to figure out what the purpose of that was. It's like reading bad code without any comments. Even if you are very familiar with the language and know what the code does, it takes a lot of effort to figure out why it's that way. If it's not explained properly.

The worst part is, no one ever complains about this stuff because they are afraid of looking stupid. I sure fear that by posting this very comment. I actually am familiar with the notation used in this example. I still find it unnecessary and exhausting to decode.

Re:Failures of Deep Learning by FridgeSeal on Saturday March 25, 2017 @04:24AM (#13954614)
May I ask what your background is? I come from a maths + stats background and the "impenetrable math notation" is literally second nature to me now: I can test and understand it with far more speed and ease than I do any programming language.

Academic papers make the very reasonably assumption that you are familiar with basic mathematical notation, and probably, the basics of your field.

It's intended audience already knows these things, criticising these papers because essentially "I don't like the notation" is a pretty useless exercise.

Side note: maths papers used to be written with words instead of notation, but we dropped that, because it was inefficient and difficult to read and understand.

Re:Failures of Deep Learning by Houshalter on Saturday March 25, 2017 @10:06AM (#13955683)
Math notation isn't inherently bad. Even in this example, it could be justified. If it was accompanied by any sort of explanation of what it meant.

No matter how familiar with the notation, it's always going to be less clear and take time to unravel it. "OK, I see the author is multiplying a bunch of numbers together? Why are they doing this? [Reads over it 10 more times]... Oh... it's to tell if it's even or odd. Why on Earth couldn't they just say that.."

And literally every use of notation in this paper is like that. I was incredibly fortunate they decided to stop at some point, and just say they then draw a line with those parameters. They could have kept going and defined a procedure for drawing lines. I would not be remotely surprised.

I believe it's entirely about signalling. Math notation looks professional and academic. Just like describing yourself as "we". There's also something called the illusion of transparency. It's a bias where people believe they are much more understandable than they actually are. Like if you explain an idea to someone, you will expect them to have a much higher chance of understanding it then they actually do. I believe people that write papers are incredibly guilty of that.

And every freaking academic paper is like this. So many papers pointlessly give exact equations for a neural network. Instead of just saying "and we created a 3 layer neural net with a softmax output trained withstochastic gradient descent." But figuring out that's what the equations describe is going to take 15 minutes and lots of confusion.

Really, imagine a programmer that explained his ideas entirely in code without any comments. With one letter, undescriptive variable names. Who used as much premature optimization as possible to obscure it further. I doubt such a post would make it to the top of HN. But this crap regularly does.

Re:Failures of Deep Learning by yorwba on Saturday March 25, 2017 @12:32PM (#13956356)
> If it was accompanied by any sort of explanation of what it meant.

Most of the paper is just text, explaining their results in plain English, interspersed with technical vocabulary.

> "OK, I see the author is multiplying a bunch of numbers together? Why are they doing this? [Reads over it 10 more times]... Oh... it's to tell if it's even or odd. Why on Earth couldn't they just say that.."

Using powers of -1 to test for parity is a fairly standard trick, and they explain it right at the bottom of page 2. Did you overlook that, were you unfamiliar with this use, or is the explanation not clear enough? Only one of these possibilities can be blamed on the authors.

Or maybe you were talking about the formulas at the top of page 4, were they are multiplying two of those random parity functions together? That's not really to check whether they are even or odd, it's to prove their mutual orthogonality, which it says in the sentence above. In these transformations, the notation using powers of -1 is very convenient, since it allows to apply simple algebraic transformations. Those would have been very tedious had they used a boolean even/odd indicator.

> They could have kept going and defined a procedure for drawing lines.

They do this: Note that as the images space is discrete, we round the values corresponding to the points on the lines to the closest integer coordinate. They don't use notation for this, because it's not required for clarity.

Basically, I think the ideas in this paper are laid out pretty clearly for someone who is already familiar with the notation and conventions used. Research papers are primarily about communicating ideas to people working in the same field, who don't need to have the basics explained to them. That this makes the results hard to understand for a more general audience, even if they are experts in their field, is unfortunate, but basically unavoidable. Adding links to definitions and examples in introductory textbooks to every research paper would be pretty awesome, but it shouldn't be the author's burden.

Re:Failures of Deep Learning by reader5000 on Saturday March 25, 2017 @12:38PM (#13956388)
The first author on this paper does have a great intro book on precisely this area where the notation ramps up more slowly.
Re:Failures of Deep Learning by soVeryTired on Saturday March 25, 2017 @06:11PM (#13957795)
But he's not testing whether it's even or odd. The lines don't necessarily pass through the origin.
Re:Failures of Deep Learning by hueving on Saturday March 25, 2017 @05:03AM (#13954711)
Great, but if what GP says is true, the math notation in this case was so terrible that it took up more space than a simple English explanation and still took longer to parse for someone versed in the notation.
Re:Failures of Deep Learning by abecedarius on Friday March 24, 2017 @09:33PM (#13953577)
I've complained about this sort of thing too (usually more privately). The upside is supposed to be precision and concision, but as you point out in this case they spent all their concision on buying precision. For this precision I think code should be considered more often instead of the usual informal math notation, which sometimes gets sloppy and harder to follow from outside the author's research community. (Edit: I'm not accusing that passage of sloppiness, but it's a problem that's frustrated me before with math in place of code.)
Re:Failures of Deep Learning by nojvek on Saturday March 25, 2017 @02:05AM (#13954319)
I will dance around the day someone figures our an AI to translate papers into readable code that compiles into a nice jupyter like notebook format.

Some papers are quite easy to comprehend and truly ground breaking e.g the CNN paper by Alex and Hinton.

Some like this are like "WTF are you even trying to say?"

Re:Failures of Deep Learning by reader5000 on Saturday March 25, 2017 @12:59PM (#13956512)
Code will be hard to follow outside of the particular language's community. There are efforts to standardize pseudo-code, but then you're basically back to math notation.
Re:Failures of Deep Learning by Tarq0n on Saturday March 25, 2017 @01:28PM (#13956638)
Code has the advantage that functions and operators are googleable, while mathematical notation is not. I think the search problem is a fairly serious issue for maths actually.
Re:Failures of Deep Learning by reader5000 on Saturday March 25, 2017 @04:08PM (#13957323)
True, but it's not maths' fault Google and OSs dont support copy/paste for notation. Also, if you have to google e.g. the expectation operator you probably shouldnt be reading research papers in theoretical machine learning.
Re:Failures of Deep Learning by abecedarius on Saturday March 25, 2017 @02:21PM (#13956884)
And the even greater plus of being runnable. If it's confusing/ambiguous to you, you're not stuck.
Re:Failures of Deep Learning by outlace on Saturday March 25, 2017 @03:17PM (#13957116)
I think mathematical formalism is necessary to communicate precisely what you're doing, however, I do wish academic papers supplemented the math with just some intuitive plain English describing what's going on.

Sometimes you just want to get the gist of what the authors are doing without having to solve math problems in your head. Other times you really are interested in actually re-producing what they did and so you really need the math/code.

Re:Failures of Deep Learning by gajjanag on Saturday March 25, 2017 @08:41AM (#13955334)
I do not agree with your point, for various reasons already elaborated upon in the comments section here.

Regardless of that, I find it very puzzling why you don't simply email the authors with your critique, instead of writing a pretty thorough comment on HN that they will likely not read? That would be far more constructive IMHO.

Keep in mind that this is a preprint. From my experiences in this field, preprints are usually very sloppy and are simply a means to lay "first claim" to a discovery. There are usually a ton of typos/other issues to be sorted out, or even worse, bugs in many key proofs - I am not saying this paper is guilty of this, just that expectations should be low regarding exposition quality in preprints.

Re:Failures of Deep Learning by chaoxu on Saturday March 25, 2017 @04:47AM (#13954667)
I do see this kind of behavior in engineering oriented fields. Maybe the authors try to be as formal as possible, but doesn't known when to stop? Or maybe this is how they naturally think? (I doubt this, but because this is so common, I wonder if this is actually the case...)

I had a similar problem in my earlier career, because I try to be as precise as possible. But for conciseness, I don't bother to explain anything. Later on I learned to say things in English, or give intuition, before writing out the formal definitions.

Fortunately, papers the reviewers cannot understand have a hard time getting into top conferences in my area (CS theory).

Re:Failures of Deep Learning by Houshalter on Saturday March 25, 2017 @09:45AM (#13955592)
>Or maybe this is how they naturally think?

I doubt this. If you watch a lecture by an academic, they will usually speak in understandable English. Even if the subject is highly technical. They will give motivation and explanation for why things are that way. It seems only in academic papers do they go to great lengths to remove those things.

Re:Failures of Deep Learning by bunderbunder on Saturday March 25, 2017 @07:13PM (#13958017)
Keep in mind that the theoretical work in machine learning tends to come from a rather different academic sphere from a lot of work in application. That means a different professional vernacular, different notational conventions, and different ways of expressing certain concepts.

I don't think it's quite reasonable to fault an academic for writing under the conventions of their field. That's not being needlessly obtuse, that's being as clear as possible for the intended audience. There's no call for sour grapes just because you happen to not be the intended audience of an academic paper in a field you don't actively participate in.

Re:Failures of Deep Learning by EpicEng on Saturday March 25, 2017 @02:01PM (#13956778)
It's not unnecessary, you're just not the target audience. I often implement algorithms documented only in academic papers. Those that lack the precision you're complaining about are often useless. I need the math.
Re:Failures of Deep Learning by soVeryTired on Saturday March 25, 2017 @05:20AM (#13954741)
I don't agree that they're redefining even vs. odd. It's more like positive sloping vs. negative sloping. The reason they've introduced the y variable instead of referring to the slope of those lines is that they're going to use the y value as a class label later.

I agree that the paper is written in dense technical language, but the point is that the authors are writing for other academics, not laypeople. They could go to some lengths to avoid using symbols like ~, but those symbols are completely standard in stats and ML. If you haven't seen them, there's a good chance you don't have the right background to follow their paper anyway.

All writing styles have their place. In textbooks and blog posts, it's completely sensible to have an index of notation, and to use natural language to express ideas. But when writing for an audience of trained scientists, it's a different matter.

Re:Failures of Deep Learning by gesman on Friday March 24, 2017 @10:45PM (#13953822)
Otherwise it won't get pass through arxiv's nonsense moderation.