> This will get messy quickly if you have a PR that you are adding commits to and have a failure for each commit. This has the same problem as some code coverage solutions, they are spamming the PR with comments that are only important for a few moments, then they are just noise...

IIRC at least Codecov (https://codecov.io) has logic to go in and update the existing comment rather than spamming with new ones. It's nice!