I have used Splunk in the past on-prem. They are expensive and to your point and a lot of capability that wasn’t used. We had investigated Gray Log, Fluentd, Logstash and others. If we started now, I’d say the ELK stack (Elasticseach, Logstash, Kibana) would be what we’d have gone with.