Useful capabilities for tracing today that could translate well for logs too. So long as everthing is stitched together, which is what these tracing libraries are responsible for.
At least with Honeycomb, for logs unassociated with traces, you can use dynamic sampling[0] to significantly cut down on total event volume and bake in good representivity. And when you have logs correlated with traces, like how OTel does it, you can sample all logs correlated to a trace when that trace is also sampled[1]. We've at least also been doing some thinking about what batching up and coherently sampling "likely related but not correlated by ID" groups of logs. It's definitely in the realm of doable and TBH I find it a little strange that people haven't sought these kinds of solutions at large yet. I think most folks still just assume that the only way you can reduce volume is by giving up some notion of representativeness when that's just not true.
But metrics are an entirely different beast.
Sure some metrics can be derived from "events". But what about thinks like CPU and RAM usage? Do you want to record an event for every packet so you can measure network performance? And as mentioned above, you generally don't want to sample the things you have metrics if.
Metrics are a sampling of a counter/gauge.
Usually, every log message occurs as part of a process (no matter how short lived), and so I'm not sure it's ever the case that a given log is truly independent from others. "Sampling log events" is a smell that indicates that you will have incomplete views on any given txn/process.
What I find works better is to always correlate logs to traces, and when just drop the majority of non-error traces (keep 100% of actionable error traces).
> When you can not store every event, you want to drop randomly at the whole trace level.
Yes, this.