If you're not changing your tests on every interview batch you're doing it wrong.

You're asking a potential candidate to spend hours cracking your program but you can't be bothered to change it for each batch?

I expect any company applying this interview technique to at least put some effort into it and even then it pays off since a batch could contain hundreds of candidates and you only have to build the program once.

This ^ "You only get out of it, what you put into it." goes both ways - employers should expect to extend some effort if they expect their candidates to extend some effort.

I strongly suspected that sometimes goals of the person responsible for hiring are not quite aligned with the company they work for. Example:

Company's goal: Hire an exceptional candidate.

Lazy or overworked hiring manager's goal: Reduce the size of the stack of 1000's of resume's I have to go through using any means possible that will make it appear like he/she actually did the work to find the candidate.

"Lemme see, use keyword search to ditch the ones without 'Github' on their resume'. Now ditch the ones who don't have a college degree. Still some resume's left? OK, how about tossing the ones without a CS degree?"

Run a filter with enough conditions and you can get that pile down to nothing pretty fast. If that is still not enough filtering, ask the candidates to do something very time consuming. This a great way to weed out people who are very busy or exceptional at what they do and don't need to jump through your hoops.

The problem is that this makes it difficult to compare evaluations. If you want to have as objective an evaluation of candidates as possible, you need to subject them to exactly the same process.

I'm surprised they didn't make him sign an NDA for it...

What do you mean by "batch"? Most companies have rolling hiring.

this is true - but you're [generally] going to have a batch of candidates for any given position

Pick a unique keyword for it all and SEO the hell out of a site selling the walk thru.

Make it slightly wrong, or obviously fingerprinted.