> I've been tempted to just ship straight to a dB and skip all these crazy shippers and parsers and all the other middle men in the equation.
You need the parsers. You want to find a needle in a haystack? Good luck without having things broken down into proper fields with metadata.
You need the shippers. Elastic Beats has full backpressure support so that when your cluster is busy, it can intelligently back off. Otherwise, you'll drop logs, or overwhelm the system to the point of uselessness....
> why has no product unified monitoring and logging?
Metricbeat from Elasticsearch + Grafana aimed against Elasticsearch to get you better dashboards and alerting.
Please don't reinvent the wheel on this one. Deploying ELK + Beats + Grafana is not that hard, there's tons of documentation, and it is a very stable product.
I haven't checked, but I bet it's cheaper than ES for data that's mostly cold, like logs. You will need a separate monitoring solution though.
What is the difference?
Can you just extract metrics from logs via ES. Logstash even has a prebuilt JAVASTACKTRACEPART for java exceptions.
Metrics can always be generated from logs, especially structured payloads.
The docs have more info about both: https://www.elastic.co/guide/en/infrastructure/guide/current...
Disclaimer: I work on Kibana at Elastic
Perhaps I misunderstand your situation, but I don't see any "CREATE INDEX" available in Clickhouse, and thus won't "SELECT * FROM logs WHERE match(message, '(?i)error.*database')" require a full column-scan (including, as you mentioned, decompressing it)? Versus the very idea of an indexer like ES is "give me all documents that have the token 'ERROR' and the token 'database'" which need not tablescan anything
I only learned about the project 9 minutes ago so any experiences you can share about the actual performance of those queries would be enlightening -- maybe it's so fast that my concern isn't relevant
If your query is linearly scalable conceptually, Clickhouse is also linearly scalable. Per core performance is also pretty good. (tens of millions of rows per second on good hardware and simple queries, like most log aggregation queries are)
If you value being able to store arbitrary log files, ClickHouse is not for you. If you want to build your system to generate tables on the fly- ClickHouse might work.
Disclaimer: I work for Altinity which is commercializing ClickHouse.
Namely InfluxDB and friends
[1] https://jumptrading.github.io/influxdb/metrics/monitoring/de...
It was built for .NET devs using Serilog but can accept structured logs/json over HTTP.
In my experience SumoLogic is excellent for this.
I've been tempted to just ship straight to a dB and skip all these crazy shippers and parsers and all the other middle men in the equation.
Also, why has no product unified monitoring and logging? AFAIK that's why Splunk is worth it is you have the budget (I dont)