I have looked around quite a bit but haven't really found anything that strikes the right balance for me of features, cost, effectiveness, etc.

I've been sort of building my own system in Go which is already a big improvement on the Python-based solutions, but it's nowhere near ready for this type of thing. But it uses Go-colly and go-readability for web scraping.