I think you generally need to either assume or establish that your testing environment tends to be sound, just like you need to assume or establish that your programming language/compiler/computing environment tends to be sound. Your application code and test code itself probably shouldn't be compensating for or attempting to avoid known or unknown flaws in your testing environment.

How do you establish that the testing environment is sound besides having defensive tests? It doesn't necessarily have to be the case that all your tests are like that.

Why not just use a hash function that special-cases two test URLs (or a whole class of test URLs) and produces collisions for them? You can use that same hash function in production.

That’s a stupid principle. If you don’t have your interface well-defined enough that switching out the hashing implementation is difficult, your code is too tightly coupled.

A test handing a hash collision is not a test of the hashing algorithm so you should be able to switch out the algorithm to something super predictable. This isn’t a discussion about mocking frameworks - you should be able to do with by just plugging in a different hash implementation when you construct the test.

These types of blanket rules are useful to ensure a 100% accurate test case. If an error in production costs me $15 Million I don't really care if the engineer wastes a few days to follow a dumb rule.

Also I can employ a $75k - $100k/year engineer and say follow this protocol for testing. If I want someone who can optimize test case 100% of the time with zero mistakes ever then I might have to pay $150k - $200k / year.

That’s not useful. A swapped out hashing algorithm would have been proving the same thing and would have allowed you to test important things like 3+ collisions and two sets of collisions.

Anyone who demanded the real hashing algorithm in this case doesn’t understand how hash collisions work and what problems you need to solve when using hashing. Your “100% accurate test case” serves very little value.

It’s the same as requiring a real person perform the checkout workflow in your shopping cart integration tests. It’s “100% accurate” but it’s a massive waste of resources and provides terrible coverage of the many other edge cases.

It ensures that if someone changes the hashing algorithm, your test for the collision mitigation now doesn't test that anymore even though they should be unrelated.

After going through all that effort to make the test less "fake", I expect tialaramex verified the collision still happened as part of the test.

Yes, actually we also verified that all the hashes were exactly what we expected, because we were using this for content based addressing. "The hash has unexpectedly changed" was a fatal build error.

If we wanted to change how the hash is calculated all extant instances of the system need to dump to backups, then be painstakingly restored after updating. Possible, but certainly not something you'd be doing for giggles.