To my knowledge nobody has really solved this problem yet in a satisfactory way, outside some closed-source solutions at big internet companies.
Our service isn't free beyond 5 hosts, but is quite a bit cheaper than rolling and running your own or flying blind and facing the consquences. It's a hard problem, and we're on a mission to solve it for companies that don't start with goo* or end in *ook.
Without some understanding of the dimensions of the data, it is very difficult to compose dashboards or aggregation rules that have arbitrary filters or can update automatically when new components are added.
I really ought to elaborate in a blog post someday :)
You can attach any arbitrary set of tags to metrics or events - on a per-datapoint basis, and slice / dice / alert based on those tags. Datadog will automatically tag your points by chef role or AWS availability-zone, for example, and you can add any other tag you want. Tags also don't have to be tied to a host and can also relate to a specific volume, mysql index, etc...
(Note to HN, this is a feature of datadoghq - pup will gladly collect and filter on tags, but won't aggregate them, yet)
Suppose I apply the tags "ord", "foo.example.com" and "bar.example.org" to a metric. How does the dashboard builder or the aggregator know what they are? They're values without the fields.
And you can then ask for "my.metric" summed over "availability-zone:us-east-1a" grouped by "role".
Here's a screenshot to show what it looks like https://skitch.com/oliveur/emywc/sobotka-datadog
You obviously have a lot of experience with monitoring tools - I'd love to connect. I'm @oliveur on the twitters.