By depending on ${pysrc} which is set above with `pysrc = $(shell find ${pysrcdirs} -name *.py)` you ignore the possibility that a file may be removed. This would happen in a bisect for example or if someone were to manually delete a file. This will prevent tests from running.
I think the bigger issue with your makefile is that there's about 5 targets in total which actually make use of any of make's features, and a lot of targets which make incorrect assumptions about how make works.
Here's a list of targets which appear to not use any of make's features at all (and no, depending on ${app} being set up before they are ran does not constitute using make's features): audit, run_no_backend, run-frontend, run-nonlocal-frontend, app, run-worker, run-broker, testcase, test_integration, test_system, test_datasets, test_deterministic, test_mysql, test_postgres, clean, clean_virtualenv, mrproper, pip-sync, _mark_outdated, _commit_update, ${python} ${pip}.
Moreover, here's an example of a target with fundamental issues:
update_requirements: _mark_outdated requirements.txt requirements-dev.txt requirements-deploy.txt _commit_update
This is just plain wrong. You're treating make dependencies as some kind of sequential list of commands. That's not how make works. If I was running this with make -j or a gnu make implementation with by-default nondeterministic target build order choice this would just break.
I'm not saying your makefile saves zero purpose. There's a few targets there which you should keep. But it should really be 90% shorter with all the removed bits in a shell script so that you're not relying on nuances of your particular version of GNU make to make it work.
That being said, while perusing the original makefile I found multiple issues. The one which stands out the most is that running make with a -j flag (which is entirely likely if your end user aliases that in their bashrc, I know multiple people that do). I am pretty sure fix and test can't run concurrently, at least I can't imagine how in-place fixing of files while running tests would work reliably. There's probably more cases where this would break things. Another issue is using `echo -e` and other bashisms, except that it's not guaranteed make will be running bash.
I really don't understand why your makefile keeps creating files, if it didn't create most of those files it would behave basically identically except for not needing to periodically run clean. In fact, by not providing a full DAG your makefile is just annoying in that there really isn't any option other than deleting random files or redoing all the steps. I don't see how having to delete .make.test and running make test is any better than having ./run-testsuite or ./run testsuite just run the testsuite every time.
Edit: I noticed your makefile sets SHELL. But this will in turn break this line:
https://gitlab.com/internet-cleanup-foundation/web-security-...