Really? I mean, to first approximation, all software engineering work is debugging. And to first approximation, all debugging is about carefully reading results and the code that generated them. That sounds pretty spot-on to my eyes.

And the variable naming quip... are you serious? One of the worst mistakes you can make when trying to read code for understanding is to rely on documentation layers like comments and variable names. Those things lie. Read the code.

I mean, obviously people do rely on those things for understanding, so when writing code it's important to have good semantically and contextually clear names for stuff (which certainly can include single letter names for local symbols repeated often!). But for debugging, you really have to train yourself to read the code, not the names.

I don't really agree that these questions test debugging skill. In real life if I read a bit of confusing code and get tricked, I will usually know immediately thanks to my IDE, compiler, automated tests, debugger, or behavior of the program. On this test if I get tricked, I just lose points.

In my experience most debugging presents as trial-and-error (a terrible test-taking strategy) or seeing a bug and reasoning at a high level which component of the software is responsible and gradually narrowing it down. Carefully reading through lines of code is usually the last step, unless I've got an especially elusive bug.

> And the variable naming quip... are you serious? One of the worst mistakes you can make when trying to read code for understanding is to rely on documentation layers like comments and variable names. Those things lie. Read the code.

I'm not sure what sorts of code you're reading but this opinion seems extreme. Do you really just wholly ignore comments and variable names? I have encountered inaccurate examples of both but I have never thought to distrust them by default. As far as I recall this has not caused me serious issues while debugging.

I don't think I agree. You're talking about heuristics for debugging code and I'd argue that many of us apply similar heuristics to reading comprehension. Computer Science is not the same thing as reading comprehension, but I have to argue that if you're really good at one of these; you're probably potentially good at the other.

Being able to go from some constructed story to a math problem is a skill that can be tested elsewhere (and really, should be assumed if someone clearly has proficient enough numeracy to do combinatorics) Questions should be testing one thing as isolated as possible, not the same things repeatedly.

I can see this in my students in my introductory class. You can ask them to calculate a concentration of a solution and they will reply just fine. But when you ask them to go into the lab and actually prepare a solution of a certain concentration you draw blanks in 9 cases out of 10. Students are trained in AP Chemistry to spot problems according to keywords and apply the appropiate formula, but they have never thought about the underlying meaning of the things they do, and they cannot connect different subjects to save their lives.

The mechanical kinesthetics of combining things is not the same skillset as the mathematical process of computing it. Lab work is its own craft.

Based on what you're saying, I'm assuming you already put your questions in the "Story time" format and are getting that 1/10 success rate because the story problems are yet a Third skill - this time one about reading comprehension and language mastery.

Reading comprehension isn't kinesthetics and that's not computation. One may partially inform the other, but each skill is fairly independent.

The sample questions cover amortization, memory allocation, finite state machines, formal logic... If you just studied enough to pass a whiteboard interview, I'd hazard that you probably can't answer half of these questions without looking up what the symbols mean.

Sure, most people here may be able to figure out the PIN question, but it's not about whether any single question is easy or hard, but whether the test can sort the students according to ability. A good student may misread one or two, but a bad student is going to miss a lot more.

What this really measures is corricilum. And pressumbly it's more stuff like this in the US then Russia?

If a CS student taking this test does not get “tricked” by this question, they should definitely be able to answer it.

Does anyone have an assessment of reliability/validity/fairness of multiple choice comparing to other test types? (afair objectivity is very good in multiple choice tests)

For a fair comparison, the translator would have to fully appreciate the relevant connotation of every significant word like "different" in every question, and successfully convey the same in Russian. You couldn't just go to a regular translation service and get a good result.

Although I have no faith in a multiple choice test as a predictor of CS knowledge anyway. That is just a horrible test format anyway.

I did not get 100%. I actually thought the exam was pretty good, but it tested something like 1% of the skills I use as a working programmer. Skills like API design, software architecture, documenting code, and yes even naming variables, which I think are important to writing quality code.

It can be easy for people to forget that many people (even coders) have never done this type of math before.

The first time you pick, you have 10 different numbers to choose from, the second time, 9, the third time 8.

1000 = all possible combinations from 000 to 999.

less 10 combinations with all numbers identical is 990.

now there are three patterns for numbers to be identical left that need to be taken out: 00x 0x0 x00. The x can be replaced with all digits != 0 in this example, so thats 3 (patterns) x 9 ('x' digits) = 27. That times 10 digits that can have double patterns is 270.

1000 - 10 - 270 = 720.

    import itertools

    print(len([x for x in itertools.product(range(10), repeat=3)])) #1000
    print(len([x for x in itertools.permutations(range(10), 3)])) #720

The reason it's less is because each number has to be different, so you can't have 000 or 111, etc.

The formula is:

n! / (n − r)!

Since there is 10 (n=10) total numbers, 0-9. And it can only be 3 (r=3) digits long. Then it would be,

10! / (10-3)!* which would result in,

10 * 9 * 8 * 7 * 6 * 5 * 4 * 3 * 2 * 1 / (7 * 6 * 5 * 4 * 3 * 2 * 1) = 10 * 9 * 8 = 720.

for j = 1-10

if i == j break

for k = 1-10

if i == j or i == k break

log(i,j,k)

(I don't understand HN formatting still)

    #!/usr/bin/python
    r = [(i,j,k) for i in range(10) for j in range(10) for k in range(10) if i!=j and i!=k and j!=k]
    print(len(r))

Do people really struggle with problems like these?

Don't use that same password for anything else!

Different can be interpreted in multiple ways.

Reminds me of a physics test where the question is something like:

Susie is almost at her house and comes to a stop in 4 seconds. What was her acceleration.

Anyway - IMO this is a terrible question.