Also IMO open-ended query language doesn't help in quick exploration, it's a barrier. UI and easiness of use is the paramount for adoption. To achieve arbitrary queries one can dump everything to smth like ClickHouse and query it with SQL. This would be a nice addition to any observability stack, to cover for a small percentage of the very deep explorations.
If you want a point-and-click interface to log searching I agree that some percentage of people like to start there (and I think splunk may even have that too), so I'm not opposed to it existing at all, but I feel very strongly that having the more sophisticated capabilities if you want to move beyond a point-click is a requirement.
To make it easier for myself, I think of spans also as structured logs with a schema that everyone had agreed on, which make it possible to trace requests across multiple services/clients. It's probably more than that, but I don't need academic precision to see how this is more useful during livesite investigations than simply querying logs with unaligned schemas.
- it highlights the intention of storing as much context as possible - it also hints on the implementation for a system that would serve them. One likely need to use columnar storage to store wide events, there is no way around it
But just a personal preference in the end.
The other option I've seen is a guided query language with helpful UI affordances as you type. I've seen it on Datadog as well as Jira advanced search.
You can trivially do that in Scuba too using a derived column, which is supported in the UI. If you need more complex stuff you can write your query in SQL and still use all the supported UI visualizations. And for even more complex stuff, for example if you need joins, the data is usually mirrored to Presto, so you can write arbitrarily complex queries and still visualize the results in the Scuba UI.
I'm not sure if your assessment of Scuba is based on full knowledge of what you can do with it.
Datadog has a new feature for querying the whole trace call graph that they seem to be testing out. Sounds like precisely the thing you're describing.
A tool like splunk can do everything scuba can do and a million things it can't. Sumologic can too.
The reason that splunk/sumologic are so much better than scuba is that they have open-ended query languages rather than this on-rails "only ever do one group-by". Just for example, if you wanted to dynamically extract a field at query-time based on the difference of two values and group by that, that's something you can do trivially in splunk/sumo.
I could write a whole essay on the topic really, but the gist of it is you need a full-scaled open-ended language for advanced querying because 1% of the time you need to do weird stuff like count-by -> a second count-by.
What I will agree with is that traces/metrics do not inherently give you this ability, but absolutely traces could if there was a platfom with a powerful enough query-language for it (e.g. give me all requests that go through 4 services, have errors on service 3 but not 4, and are associated with userId 123 on service 1)