If you’re using pytest you just paramaterize the tests and it tells you the exact failing case. Seems to be a basic feature I would be surprised to know doesn’t exist across almost all commonly used frameworks.

We are using unittest.

But thanks for bringing it up, it seems it also has "subtest" support, which might be easier than interpolating the error message in some cases:

https://docs.python.org/3/library/unittest.html#distinguishi...

There is also `parameterized` [1] if you want to do it by using decorators.

[1]: https://pypi.org/project/parameterized/

> We are using unittest.

Unless you need the library style in order to drive it, switch to pytest. Seriously.

- assert rewriting is stellar, so much more comfortable than having to find the right assert* method, and tell people they're using the wrong one in reviews

- runner is a lot more practical and flexible: marks, -k, --lf, --sw, ...

- extensions further add flexibility e.g. timeouts, maxfail, xdist (though it has a few drawbacks)

- no need for classes (you can have them, mind, but if functions are sufficient, you can use functions)

- fixtures > setup/teardown

- parameterized tests

You mean "subTest", I switched to pytest just to get away from the camel (and it's in the standard library!)

Came here to mention unittest.subtest. It's for exactly this case.

This seems like a test design issue to me. Best practice is to avoid for-each loops with assertions within tests - using parametrized tests and feeding the looped values as input is almost always a better option. Figuring out which one failed and why is one advantage it gives you in comparison. Another one is that all inputs will always be tested - your example stops on the first one that fails, and does not evaluate the others after that.

not really. one thing this is useful for is extracting out various attributes in an object when you really don't want to compare the entire thing. Or comparing dict attributes, and figuring which one is the incorrect one.

for example,

    expected_results = {...}
    actual_obj = some_intance.method_call(...)

    for key, val in expected_results.items():
        assert getattr(actual_obj, key) == val, f"Mismatch for {key} attribute"

You could shift this off to a parametrized test, but that means you're making N more calls to the method being tested, which can have its own issues with cost of test setup and teardown. With this method, you see which key breaks, and re-run after fixing.

Ok, in this case a parametrized test is not the best approach, I agree. But I would still want to avoid the for-each and "failing fast". One approach would be to gather the required attributes in an array or a struct of some sort, and then do a single assert comparison with an expected value, showing all the differences at once. However, this requires the assertion framework to be able to make such a comparison and return a nicely readable error message, ideally with a diff.

Right, and not many actually do. with python and pytest, you could leverage difflib, but that's an additional thing that adds unnecessary complexity. My approach is simple enough, good enough, and doesn't require additional fudging around with the basics of the language's test libs.

also,

>your example stops on the first one that fails, and does not evaluate the others after that.

I would argue this is desirable behavior. there are soft checks, ie, https://pypi.org/project/pytest-check/, that basically replace assertions as raised exceptions and do your approach. But I do want my tests to raise errors at the point of failure when a change occurs. If there's alot of changes occurring, that raises larger questions of "why" and "is the way we're executing this change a good one"?

> your example stops on the first one that fails, and does not evaluate the others after that.

I don't think this is a big problem; trying to focus on multiple examples at once is difficult.

It might be a problem if tests are slow and you are forced to work on all of them at once.

But in that case I'd try to make the tests faster (getting rid of network requests, disk/DB access by faking them away or hoisting to the caller).

Hm. I think my main issue there is not the speed, but rather seeing the whole picture at once. You mentioned you use this pattern to test regular expressions; say you modify the regexp in question with some new feature requirement, and now the very first of a dozen test inputs fails. You fix it, but then each one of the following keeps failing, and you can only find an elegant solution that works for all of them after seeing all the failures, having ran the test and modified the code a dozen times. Wouldn't it be nicer to see all fails right away and be able to find a solution to all of them, instead of fixing the inputs one-by-one?

In my experience, from doing some TDD Katas[0] and timing myself, I found coding slower and more difficult when focusing on multiple examples at once.

I usually even comment out all the failing tests but the first one, after translating a bunch of specifications into tests, so I see the "green" when an example starts working.

Maybe it would be easier to grok multiple regex examples than algorithmic ones, but at least for myself, I am skeptical, and I prefer taking them one at a time.

[0] - https://kata-log.rocks/tdd

In my own experience, this has often been a good way of going in circles, where I end up undoing and redoing changes as fixing one thing breaks another, until I take a step back to find the proper algorithm by considering multiple inputs.

Of course, ymmv depending on how good your initial intuition is, and how tricky the problem is.

Again pytest makes things so much nicer in this regard. Having to comment things out sucks.

With pytest you can use the -x flag to stop after the first test failure.

Even better you can use that in combination with -lf to only run the last failed test.

> With pytest you can use the -x flag to stop after the first test failure.

> Even better you can use that in combination with -lf to only run the last failed test.

Fwiw `--sw` is much better for that specific use-case.

`--lf` is more useful to run the entire test suite, then re-run just the failed tests (of the entire suite). IIRC it can have some odd interactions with `-x` or `--maxfail`, because the strange things happen to the cached "selected set".

Though it may also be because I use xdist a fair bit, and the interaction of xdist with early interruptions (x, maxfail, ...) seems less than perfect.

Oh nice, didn't know about that flag!

Another option is to use a custom mark on the test you want to run, and then do something like "pytest -v -m onlyrunthis"

I think we also have to take into account how you work.

Are you working by yourself? Nobody else will ever see this code? Do whatever your timings tell you is best.

Are you working in a team or is this an open source project? In that case I think we need to do the thing that allows for the most variation. People that like having an overview of all the failures at once can only do so if you don't fail on the first error. If you do they have no workaround. If you show all of them though then your workaround is to simply only look at the first failure, ignoring the others.

Personally I do a hybrid approach. I want my test suite to continue through all tests and tell me everything that is wrong. If I change a small thing over here I definitely want to see whether I broke 3 or 4 unit tests around there or if I broke 37 test all around.

For fixing them I will go one by one after having done a quick survey of the types of failures. I. e. I might expect some of those failures due to the nature of my change. IDEs are really great here at least in my world. They allow me to very quickly run either a single test, a certain subset or the entire suite. After the one test I concentrated is green running all tests in that particular file again is just a click away. Now I am in overview mode again. I want to know what overall effect this one change had. Everything green again? Different subset broken? No change?

That looks like a case where I would use a parameterized test rather than a for loop inside the test.

this has downsides if you're comparing attributes with a method result and checking whether said attrs match what you expect. Either you run each test N times for N attr comparisons, accepting the cost of setup/teardown, or do a loop and fire off an assert error with text on which comparison failed.

Since you already have the object right there, why not do the latter approach?

If the setup/teardown is expensive I would do it in reusable fixtures. The reason I wouldn't choose the latter approach is that it would usually be less convenient in the long run. You'd need to replace your asserts with expects to avoid it throwing on the first error (if this isn't what you want), you'll often need to manually add data to the assertion (as GP did) that you would otherwise get for free, and you'll need to look at the assertion error rather than the test case to know what actually failed. This can be quite inconvenient if you e.g. export your test results in a CI/CD pipeline.

normally in a CI/CD pipeline, you'll see what asserts failed in the log output. Github Actions with pytest shows the context of the failed asserts in the log output. TBH, thought this was standard behavior, do you have experience with a CI pipeline that differs?

All the other points you make as negatives are all positives for me. Biggest thing is, if you're making this change that alters things so drastically, is that really a good approach.

Also, fixtures aren't magic. If you can't scope the fixture to module or session, that means by default it runs in function scope, which would be the same thing as having expensive setup/teardown. And untangling fixtures can be a bgger PITA than untangling unexpected circular imports