News for hackers. Stuff that matters.
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by rladd on Friday October 25, 2019 @12:45AM (#21351971)
My mother got a perfect 800 score on the GRE English test many years ago when she wanted to go back to graduate school after her children were grown up enough (highschool/college age).

She told me that the way she got her perfect score was by realizing when the questions were wrong and thinking of what answer the test creators believed to be correct.

She had to outguess the test creators and answer the questions wrong -- in the "right" way.

This seems like a similar situation.

Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by Huppie on Friday October 25, 2019 @02:06AM (#21352317)
I've had the 'pleasure' of taking some 'Microsoft certifications' at various companies I worked at in the past and this sounds extremely familiar.

"I probably won't ever do it like that and/or there's a syntax error in all four of the answers... but this is the answer you want to hear. It's wrong, mind you, but it's what you want to hear."

Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by justinclift on Friday October 25, 2019 @01:56PM (#21357508)
Reminds me of the 1 question I got "wrong" on a DOS test (years ago) at TAFE.

The question was "How do you delete all files in the current directory?". Using DOS 6.22 (I think, it's from memory).

My answer "del." was marked incorrect. Because the teacher didn't know enough about DOS to understand that's the standard shortcut for "del .". And the teacher refused to even try out the command, lets alone fix the incorrect mark. sigh

Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by jeremyvisser on Saturday October 26, 2019 @07:25PM (#21366952)
TAFE anecdote time!

In my TAFE class, I was asked to list two examples of operating systems.

I listed Linux and eComStation. The teacher had never heard of eComStation and marked me wrong.

Refused to correct my mark even when I proved him right. I'm still bitter about it a decade later.

Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by justinclift on Sunday October 27, 2019 @12:49AM (#21367934)
Swinburne TAFE as well? ;)
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by zerkla on Friday October 25, 2019 @05:26AM (#21353235)
Yep! You have to do away with conventional logic and ask yourself "What insanity would Microsoft recommend I do?"
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by Huppie on Friday October 25, 2019 @06:32AM (#21353553)
It's not always insanity, sometimes just sub-optimal / way over-engineered in my opinion.

They're getting better at it though. More recently I've done their devops certification and it looks like they're recommending somewhat more sane practices now...

There were still questions where even after three or four tries at certification / reading up on whatever Microsoft thinks is 'good' we didn't find 'the correct answer' according to Microsoft though... ¯\_(ツ)_/¯

Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by zerkla on Friday October 25, 2019 @06:38AM (#21353580)
Yeah, that's true. It's still a good idea to get an idea of what a desired answer would be, which is why those answer dumps are so popular.
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by z3t4 on Friday October 25, 2019 @08:16AM (#21354195)
I'm a spatial thinker, and I got a similar problem, I see all answers as correct. Eg. which one follows this sequence, and I can find a pattern to all alternatives. And I have to figure out which option the test author think is correct.
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by Double_a_92 on Friday October 25, 2019 @07:50AM (#21354023)
Sometimes the questions are also just broken. I.e. asking you to select the things that do not apply, but the answer would have been to opposite.
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by Huppie on Friday October 25, 2019 @08:22AM (#21354231)
Back when I took the 'C# certification' (70-483 I think?) there were multiple questions in the style 'which of the following answers will make the program compile', where all four answers had a syntax error, or the program had a syntax error at a different line that would cause an issue regardless of your answer.

I tried the dispute process but it's basically impossible to dispute / report broken questions unless you have a photographic memory.

Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by qubex on Friday October 25, 2019 @06:07AM (#21353438)
I have achieved similar results by similar means in both English and certain other subjects wherein one would assume a “true academic” would “know better” (picking out Sin[x]=2 as being “evidence of error in prior working” when x could merely be Complex, or marking “f[f[n]]=-n as “unsolvable” when it’s just requires a bit of lateral thinking). This always depresses me, like when (as a Brit) I hear Americans say “I could care less” as an indicator of disregard, when actually that indicates they are somewhere above the point of minimal regard.
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by chalupaman on Saturday October 26, 2019 @03:49AM (#21362353)
“I could care less” is sarcastic.
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by qubex on Sunday October 27, 2019 @07:04AM (#21368998)
No, it’s lax.
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by conjectures on Friday October 25, 2019 @05:09AM (#21353161)
This does seem like the meta solution to most tests, particularly standardised tests :)
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by qubex on Friday October 25, 2019 @06:27AM (#21353529)
Paraphrasing Simonyi: “Any test you can pass, I can pass meta”.
Re:Google T5 scores 88.9 on SuperGLUE Benchmark, approaching... by saagarjha on Friday October 25, 2019 @05:51AM (#21353354)
That's how I got through the SAT…