News for hackers. Stuff that matters.
Re:Recent Advances in Natural Language Processing by wrs on Sunday August 16, 2020 @05:26PM (#24181474)
I would be interested in hearing the results from humans presented with adversarial answer options. You may say that a machine learning correlations between words isn’t really learning science, but I wonder how many human students aren’t either, just pretty much learning correlations between words to pass tests...
Re:Recent Advances in Natural Language Processing by FiberBundle on Sunday August 16, 2020 @05:45PM (#24181624)
They do give an example of a question, in which the model chose an incorrect answer in the adversarial setting:

"The condition of the air outdoors at a certain time ofday is known as (A) friction (B) light (C) force (D)weather[correct](Q) joule (R) gradient[selected](S)trench (T) add heat"

I assume this might be characteristic for other questions as well, although I don't know anything about the Regents Science Exam and whether there are multiple questions about closely related topics.

Re:Recent Advances in Natural Language Processing by taneq on Sunday August 16, 2020 @08:16PM (#24182532)
That’s a terribly worded question anyway. Of the original answers, ‘weather’ is the least worst but it’s still vague.
Re:Recent Advances in Natural Language Processing by mannykannot on Sunday August 16, 2020 @09:36PM (#24182996)
It is a well-worded question for its purpose. The whole point is that, of all the options given, only one is justifiable (and it does not require a tendentious stretch to justify it, either.) Even “light” (which was not chosen) only applies half the time, on average. This is a valid test of natural language understanding.
Re:Recent Advances in Natural Language Processing by rvense on Monday August 17, 2020 @04:17AM (#24185142)
Remember when IBM went on Jeopardy? There was a question about which Egyptian pharaoh. A human with some knowledge of history might mix up Ramses and Seti, or whatever, or just not know the answer, but know that they didn't know. Watson answered "What are trousers?" with supreme confidence.

Jeopardy is fun and games and it was great for the blooper reel, but they're trying to sell this stuff to diagnose cancer and guide police efforts. Failure modes are kind of important.

Re:Recent Advances in Natural Language Processing by jacobwilliamroy on Sunday August 16, 2020 @06:03PM (#24181748)
Most multiple choice math problems can be completely circumvented by simply finding the digital root of the expression in the problem. I was surprised to find this to be true, even on college entrance exams.
Re:Recent Advances in Natural Language Processing by flir on Sunday August 16, 2020 @06:06PM (#24181770)
Seriously? Can you explain the mechanism? 'cos that sounds like - well, numerology, to be honest.
Re:Recent Advances in Natural Language Processing by jacobwilliamroy on Sunday August 16, 2020 @07:04PM (#24182141)
The digital root of the problem expression will be the digital root of the answer, because the answer IS the problem expression. They're the same number, just with all the pieces jumbled up. Comparing against digital roots of answer choices will usually eliminate most of the wrong answers. Any remaining wrong answers tend to be obviously wrong. 2 + 2 != 13 obviously.
Re:Recent Advances in Natural Language Processing by plutonorm on Sunday August 16, 2020 @07:41PM (#24182365)
Can you give an example? I am intrigued, although suspect you are making a joke!
Re:Recent Advances in Natural Language Processing by jacobwilliamroy on Sunday August 16, 2020 @08:47PM (#24182696)
You can check the properties of digital roots on wikipedia:

https://en.wikipedia.org/wiki/Digital_root#Properties

Example from ASVAB practice math test:

(x+4)(x+4) =

A. x^2+16x+8

B. x^2+16x+16

C. x^2+8x+16

D. x^2+8x+8

Since we don't have to solve for X in this problem, we can just assume x is 1, which would make the digital root of the problem expression 7. Assuming x = 1, the digital roots of A, B, C and D are 7, 6, 7 and 8 respectively. C is the answer because its last term is the square of 4.

Re:Recent Advances in Natural Language Processing by plutonorm on Sunday August 16, 2020 @09:14PM (#24182846)
Perhaps I am being dumb, but I don't see what the digital root brings to this.

You can do the same thing without the digital root. substitute 1 into all the possible answers gives: A: 25 B: 33 C: 25 D: 17

Substitute 1 into the question gives 25. Therefore only A and C are possible answers and it must be C because 4*4 = 16

This is the same process as your answer but without the digital root!

Re:Recent Advances in Natural Language Processing by taneq on Sunday August 16, 2020 @11:17PM (#24183526)
Agreed. The real trick here is knowing that, since we don't need to find x, we can sub in an arbitrary x and do the comparison numerically instead of algebraically.

Of course there's a tiny chance that one of the other equations would be equal at your arbitrary x value. If so and you get two 'right' answers just try again with a different x.

Re:Recent Advances in Natural Language Processing by benibela on Monday August 17, 2020 @09:00AM (#24186785)
It is the DeMillo-Lipton-Schwartz–Zippel lemma

If two polynomials evaluate to equal numbers with random variables values, the polynomials are almost certainly equal themselves

Ofc, x = 0 and x = 1 are hardly random

Re:Recent Advances in Natural Language Processing by jacobwilliamroy on Sunday August 16, 2020 @09:54PM (#24183108)
I suppose such is possible without digital roots. Let's try the same problem, but this time we'll multiply our constants by 3,571:

(x + 14284) (x + 14284)=

A: x^2 + 204032656x + 28568

B: x^2 + 204032656x + 204032656

C: x^2 + 28568x + 204032656

D: x^2 + 204032656x + 204032656

The digital root of the problem expression is 4. The digital roots of A, B, C and D are 4, 6, 4 and 3 respectively. The digital root of the square of 14284 is 1, and the digital roots of the last term of A and C are 2 and 1, respectively. The answer is C.

Re:Recent Advances in Natural Language Processing by dragonwriter on Sunday August 16, 2020 @09:59PM (#24183137)
Of course, it is quicker just to multiply and add the last digit of the constant to find the only possible answer (or count digits to get an order of magnitude estimate, which will also show C as the only possible answer).

When a problem of this form uses big enough numbers to warrant a shortcut, digital root is almost always a suboptimal shortcut.

Re:Recent Advances in Natural Language Processing by jacobwilliamroy on Monday August 17, 2020 @02:11AM (#24184465)
Yes you are the heuristics master!
Re:Recent Advances in Natural Language Processing by callmekit on Sunday August 16, 2020 @09:49PM (#24183083)
Maybe is case of large numbers you can compute digital root easier than the answer.

For example, 25 * 26 = 650, digital root 2. Or take digital root of 25 and 26 first, get 7 * 8 = 56, same digital root 2.

Re:Recent Advances in Natural Language Processing by flir on Sunday August 16, 2020 @08:54PM (#24182740)
I can't decide if I've just seen magic, or conjuring. I'm gonna have to go away and think about this.
Re:Recent Advances in Natural Language Processing by dragonwriter on Sunday August 16, 2020 @08:51PM (#24182714)
This is true, but in many cases amounts to reducing the problem to a more time consuming problem, though one that requires less knowledge. On timed tests, this can be a big negative.