If you're going to pick one language for the quiz (python) it seems a little unfair to non-python developers to focus the questions on idiosyncrasies of python. Whereas if all questions were of the more general algorithm variety (like you see toward the end), non-python developers could infer the answer without any esoteric python knowledge.

We talked a bunch about this. Python and JS are the two most common languages we see on our site, and Python has slightly more compact syntax, which works well in the questions. The idea, however, was not to require any python-specific knowledge. Which question do you think is looking at syntax?

> The idea, however, was not to require any python-specific knowledge

But the quiz specifically mentions on multiple answers that "In Python, unlike language X, Y, Z" OR "In Python, like [similar language] A, B, C"...I mean this says nothing about my programming ability, this says whether I know the quirks of the language in the same way that I can tell you all the quirks of JavaScript but not in Python. This to me is not a programming quiz, but a Python quiz designed to determine how much someone knows about what can go wrong in Python.

Well, the questions get into other things too.(Character encoding, timing attacks, graph algorithms.) Writing a good single-paragraph description of iterative deepening depth-first search was a challenge!

Yeah except that character encoding question assumes I know how Python handles bytearrays. That trick answer of "it will break" threw me for a loop because I needed language-specific knowledge about the types that Python could handle for that input.

And then with timing attacks - I mean SQL Injection is a fair web security question to ask someone, but the timing is like, very fringe knowledge. Most devs let their framework handle all of those kinds of things. Not sure why I would need to know this unless I was specifically in the sec world.

The last 3 questions were actually general CS because they went through search trees and shortest paths algos, which are language-agnostic.

Hey, not the GP, but just glancing through the quiz - Question 3 focuses on the fact that Python passes things by reference to functions. This is definitely specific to the two languages you guys use (as is your prerogative), but if you're expanding this for others, you should definitely explain that these focus on the JS and Python languages.

Additionally, question 2 amounts to "what is the usage of python's queue class", question 4 requires knowing whether array indexes start at 0 or 1 in python, and question 8 amounts to "what is the usage of python's bytearray class".

I'm not sure that's fair about question 2. It's hard to say because I know python (but not the deque class), but none of the incorrect answers seem like they'd be correct in any sensible language.

Like, whatever language you're using, you need to push the children of the current node on to the queue. Only one answer looks like it could reasonably do that.

Yeah, agreed. Question 2 as asked demanded Python knowledge, but once you checked against the answers provided it seemed much clearer that 3 options were invalid.

Also, strictly speaking, Python is not "pass by reference" as the term is used for other languages. It's more accurately "pass by object" or "pass by object reference." Somewhat of a pedantic distinction, and the quiz question is clear regardless.

[1] http://robertheaton.com/2014/02/09/pythons-pass-by-object-re...

[2] http://foobarnbaz.com/2012/07/08/understanding-python-variab...

[3] https://jeffknupp.com/blog/2012/11/13/is-python-callbyvalue-...

Good point. I think that objects are passed by reference in a high enough % of language in common use (PHP and C++ are the two exceptions I can think of) that this is a fair question. But if enough people think it's bad, we'll remove it.

You got me on that too. A high enough % of the languages pass stuff by value (C/C++), copy on assignment (C++, D, I'm sure there are more), or do not have the concept of references or changes to start with (Prolog, Haskell).

Anyway, if you want to bias it into Python, go for it. It's a fun quiz.

Copy on assignment was what got me, and I didn't feel terribly bad for missing it. No harm done though, seeing as nothing was riding on the quiz.

Every functional programmer in the world is looking at you askance.

Q2: THis isn't a syntax thing, but it does require knowledge of python queue objects to get right

Q3: This requires knowledge around the semantics of argument passing, which vary from language to language.

Q4: this again requires some understanding of how values are stored. Some languages will copy, while others will use a reference to the originally inserted value.

Q5: python closure syntax. python scoping rules.

Q8: unless you know what "bytearray.decode" does, the best you can do is make an educated guess from the context.

they didn't say syntax, they said Python idiosyncrasies. Knowing move/copy rules in python counts as that. It's something that isn't intuitive unless you know python, ex: the matrix question. An experienced dev who isn't a python person would pick the intuitive choice, see the matrix is malformed, then Google how to clone objects in python.

On the other hand, how many languages do you know where arrays are passed by value?

My primary language (Erlang) does. I assume all functional programming languages do.

That's true. I suppose my next argument is that seeing matrix.append(row) on its own line signals strongly that this is not a functional language.

Is that true? I thought most functional programming languages did call-by-reference. I mean, if you have immutable data you can't tell the difference (outside of performance).

Under the hood it's probably by reference, but the semantics are by value. By reference indicates that the original value can be changed in the called function.

I think. Not a language lawyer.

lol, the same mistake can be made in a lot more languages~

I used many programming languages - C, C++, Java, Ruby, Elixir, C#, Swift, Objective-C & Javascript.

I've never used Python.

I only did the first 3 , but both 2 and 3 focus on python-specific names (extend(), queue) and pass-by(reference/value).

popleft is shift in many other languages.

Not all languages manipulate arrays in the same way re ref/value.

Not all languages have the concept of range() - you could have used a simple array to be more language agnostic.

(I'm sure I could have found more, but I'm Python isn't a language I spend much time in)

Just to be fair, i think the comments in the source code has explained it well

Yes, but finding answers required to understand what was set by ref, and what was set by value, which wasn't in any comments.

It feels like a lot of this quiz is just trying to "catch me out" on subtle behaviours of Python I didn't know or might have forgot about. I guess this is fine if you're making a little trivia toy to distribute to Python developers on the internet, but I don't really think it achieves much as a hiring tool. Getting questions on this test wrong would probably demoralise a lot of great candidates.

I got them all right, and I agree that there were many gotchas. For example, the ones about the matrix, the array modification, etc. The only thing worse would have been something like:

https://www.pastery.net/mbeghh/

(it'll print 3)

Hmm. We included those issues, because we see a lot of programmers get stuck on exactly those two bugs! (In all sorts of languages, not just Python). I guess maybe that's the definition of a gotcha -- a thing that a lot of programmers get stuck on? In any case, I don't think it's an arcane gotcha! Being clear about references and shallow / deep copies is important.

Being clear on reference vs value is certainly important, but you lose the ability to honestly call it a programming quiz. Those questions make it a Python quiz - which is fine! But not what you've sold it as.

In that case, by far the biggest Python gotcha is:

    def do_stuff(mylist=[]):
        mylist.append("thing")

Which will lead to the list containing N items after N invocations (i.e. the second function call won't start with an empty list).

Yup, we see that issue too. But in that case, I think it's python specific (how python handles default values). Ruby, for example, does not have that behavior.

These issues might have a place in technical interviews, but framing it as a question the user can get "wrong" feels a bit like you're setting a trap. In a real interview, this would just be a talking point. What would be interesting to an interviewer would be seeing how quickly the interviewee grasps what's happening that is unexpected, and why. Marking the candidate down because they didn't know about these quirks beforehand is kind of missing the point.

do they get stuck on it in quizzes or actual programming? I haven't seen devs, other than new devs, get stuck on this, very rare to see it on stackoverflow other than for students. I think the problem is the lens you are seeing things through is this kind of quiz like thing and what trips programmers up doing a quiz. I'd like to see repeatable research into correlations between a vetting quiz and actual capability of producing software.

Fun exercise. Thanks for sharing.

I'd suggest fine-tuning the question about hashes. My first impression from reading the code, specifically the name user_submitted_hash, was that we were looking at some sort of client-server system where the hash was being calculated remotely and then submitted instead of the original credentials, and that this was a question about not trusting the client. Even without that, in a client-server system the attack you give as the mostly likely wouldn't make much sense given randomness in network latency, so the misunderstanding about the context still breaks the question. Maybe a little more context about how this hypothetical function would be used would help here?

In terms of the presentation here, if you want to show the answers immediately for each question, you might also like to provide some sort of confirmation button. It's frustratingly easy to hit the answer next to the one you intended, particularly on mobile devices. If you're also keeping statistics based on earlier attempts by other people, this is probably distorting your numbers as well.

Timing attacks like this still work over a network, you just have to make an LOT of duplicate requests, and average the times. The lucky thirteen attack (https://en.wikipedia.org/wiki/Lucky_Thirteen_attack) is a real example.

Good point about the accidental answers. I'll look at that.

"A lot" is going to be a mighty big number in this case, though. In some networks it will surely be prohibitively large. But even if the latency is predictable enough for that not to be the case, what real world system is going to let someone make that many failed authentication requests without doing something about it? You might be better just trying to brute force it.

The trouble with hypothetical security questions is that it's so difficult to conceive a scenario where you have a deliberate vulnerability that you want someone to see, yet no other implausible aspects that will throw off anyone capable of seeing it. I am reminded of a murder mystery party I once went to with a group of mathematicians, programmers, and other logical people. When we got to the end of the evening and the true murderer was revealed, I think the majority of the participants had long ago ruled out that suspect on the basis of one of several subtle deductions from the clues provided, none of which had apparently been intended or considered by the authors of the scenario. Instead we were supposed to have ignored all the minor inconsistencies in clues and ambiguities in phrasing, and just gone for the person with the flashing neon sign over their head in the first place...

I am actually curious to know of cases of insecure string comparison vulnerabilities being exploited. I think there are real cases. But I tried recently to to this against a toy server over my local network, and could not get it to work without adding a delay to the comparison.

Question 9 appears to have two answers that are correct, but only one is accepted.

Hmm. Just checked again. Two of the expressions match neither word, one matches both, and one matches only one.

EDIT: I see. The confusion is over partial match vs whole match. I thought whole matches were standard when talking about regular expression in the abstract, but looks like I was wrong. Just changed the problem to include the anchors. Thanks for pointing this out.

That's not the standard meaning in my experience. Regex's are often used for extracting information from larger strings, forcing a full match by default would be pretty counter to their purpose.

Yep, I just came here to post that:

In [2]: re.search("b[l].e", "babel")

In [3]: re.search("b[l].e", "blabber") Out[3]: <_sre.SRE_Match at 0x7f2db79ed030>

It looks like they meant "match" as in "re.match", i.e. "^b[l].*e$", which is not the usual definition of the word.

That would also make this question specific to python, which wasn't at all what I was thinking about. It's a question about regular expressions, it should be as agnostic as possible to implementation.

Yeah, agreed, although there are different RE flavors, so you can't get too agnostic.

To be honest, this quiz makes me feel that I'm learning more about you than you are about me. I especially find the reactions to the complaints interesting. This response could be read in a negative way, which is not as I intend it. Hopefully it can provide a window to view things from a different perspective.

It is not possible to really determine the ability of a programmer in an interview -- the scope is too large. It takes time and effort to understand how well a person will perform. In an interview we try to find indicators of how well a person will perform. So, a quiz that tries to unearth those indicators is interesting.

Your quiz makes me feel that technical details are foremost in your mind when you asses performance. Will the candidate make technical mistakes that you have seen many times before? Similarly, there appears to be considerable bias evident in your selection of questions. All of these questions asses whether or not a candidate deeply understands the functioning of the code snippet, or whether they understand it shallowly. For the candidate who admits a shallow understanding, their only recourse is to guess (and potentially fail).

From that I infer that your cultural makeup is one where you have a bar that you expect all candidates to surpass. This bar is single dimensional, though, and especially your responses to the complaints imply a lack of realisation that you are narrowly optimising for a single useful ability.

Finally, the quiz is set up in a "run the guantlet" fashion. Can you avoid the pitfalls that others have fallen into? This tells me that you fear candidates who do not statically measure up to your bar more than you are excited to find out what a candidate brings to the table.

While you clearly will have other interview techniques for other aspects that you value, this particular quiz would worry me as a candidate. It's essentially the same question 15 times in a different context. When people complain that the quiz is too Python specific, the answer is essentially "But it covers concepts that are similar for many languages and you should be able to figure it out". It's a kind of "Well, I'm sure I could do it pretty easily, so you should be able to too". It makes me worried that there is a lack of empathy on team and that there will be problems if I don't think exactly like the leaders on the team.

On the other hand, this may be fine if that's what you want in your interview process. Personally, I probably would not apply to your company having seen this quiz. This may also be a good thing from your perspective :-). However, if you ever get to a point where, despite having excellent people on staff, you can't seem to get to the next level, I would concentrate on examining potential biases of what is "good" on your team.

Good luck!