This is what the Python test framework Pytest does, among many other similar useful and magical things. I believe that the Python developer ecosystem as a whole would be substantially less productive without it.