I did something similar for a self-hosted github enterprise installation:
* A webhook would be fired at my host when a new pull-request was created, or updated.
* When the webhook was received we'd store details of the repository, the branch-name, and the PR-ID into a queue, from the webhook paylad.
* A bunch of worker machines would poll the queue, and when a new message was received they'd checkout the repository, run "make test", and add the output of the run as a comment to the PR.
This was done as a temporary thing, rather than spinning up Jenkins/similar, with the expectation we'd get Github Actions coming to Github Enterprise in the near future.
Taking this approach has two big upsides. First, the bot is just a binary running on a cheap VPS; so I know that it'll be fast and that I can ssh in at any time to debug if necessary. Second, if there's a feature that I want, I can just add it, rather than twiddling my thumbs. For example, I noticed that I was manually deleting my merged PR branches every time, so I added a few lines to the bot and now it deletes them automatically.
The obvious downsides of this approach are: 1) Implementing even an MVP of such a bot can consume significant time and energy, which you may not have a lot of; 2) If there's a bug, it's your fault and now you need to spend more time and energy tracking it down; and 3) Your bot might have a security vulnerability that exposes secrets, allows injection attacks, etc.
I'd love to see tooling/frameworks that make it easier to create custom CI systems. I think startups (and perhaps larger companies as well) could benefit a lot from building a CI in-house, since it allows you to optimize for your own specific needs. I see a lot of parallels to code linters, where spending a bit of time writing custom lint rules or static analysis tools can have a large payoff.