olidb2

Shameless plug here - The author of the article does build time-series databases for a living, and more specifically the Datadog monitoring platform - which will gladly collect your millions of metrics, graph them and alert on them, along with all the events you care to keep :-)

We've been through a number of data stores ourselves, starting with Postgres back in 2010 - then on to Redis + Cassandra before we built our own. But that's a story for another post...

http://datadog.com

OP works on the time-series processing backend at Datadog http://datadog.com

Datadog - New York City

We're hiring engineers that specialize in one of the following: + Data visualization in the browser + Go and distributed systems + System programming and low-level performance optimization + Data Engineering on the hadoop/spark/python stack

We're one of the very few NYC startups to be engineering-driven and working on hard, large-scale data analysis and visualization problems. We're also growing very fast :-)

Our product is built by engineers for engineers which means you can and will have a lot of impact on it.

More details at http://jobs.datadoghq.com/

I live across of it, and I have to say it has the grace you'd expect of a 60's housing project in the eastern bloc. The rendering is quite flattering.

I work at Datadog - we're only using ElasticSearch for full-text structured events, not time-series, which represent 10,000 - 100,000 times more data in volume.

We had to build our own Time-Series streaming / storage / query so we could handle millions of points per second and years of retention.

(we love ElasticSearch, though)

We don't use the recommendation engine but the underlying platform, which makes it really simple to write and run pig jobs. Though the majority of our business deals with real-time data processing, the ability to crunch numbers in batch without dev or ops overhead is attractive and well worth the price to us.

FWIW we've been using the mortar platform to run large pig jobs without a fuss at http://datadog.com and we've been very happy with it. Glad to see them contribute their recommender code too.

Datadog - see http://datadog.com

We're looking for engineers passionate about either

  * Data visualization using D3
  * Large-scale real-time data processing in Go, Python and C
  * UX/UI Design and development 
  * Writing technical pieces you'd want to read

Pretty sure Datadog out-purples y'all http://datadog.com

We do, but it only makes sense for very large deployments. Happy to discuss - oli@ our domain.

New York - Datadog http://datadog.com

We're a growing, well-funded startup. And we're hiring engineers for all positions from back-end to ops to data visualization.

Think tens of billions of data points a day to stream, store, visualize and analyze. We use Python, Numpy, Cython, D3, JS, Cassandra, Postgres, Redis, Elasticsearch, and more...

Check our jobs out at http://jobs.datadoghq.com

Hey Nick - come and visit us at Datadog http://datadoghq.com We're on 26th and Broadway, next to a great many other startups. I'm oli@

Datadog hearts PagerDuty. We use it every day. And sometimes nights too...

And here's its awesome API in action in Datadog -http://www.datadoghq.com/2012/07/pagerduty/

Nope - all the data stays there, and can be shown on dashboards. We just don't show the hosts that don't submit data in the host lists, and don't count them towards the total number of hosts charged for. Does that answer your question?

Pup will graph all the metrics Datadog does - it already collects OS metrics, and comes with a set of plug-ins for connecting to common apps, and it wouldn't be very difficult to connect collectd to it.

Pup's feature-set is limited today, though, compared to graphite. You may also check-out the full http://datadoghq.com service for more. It's SaaS, and free up to 5 servers.

They'll disappear automatically when you stop sending data about them (within a few hours) - we've done that for our customers who have elastic / rolling servers on AWS for ex.

If that doesn't cut if for you, we're happy to help - let us know what you're trying to do!

Got it. Tags can be of the name:value kind. for ex: role:db or availability-zone:us-east-a1 or foo:bar

And you can then ask for "my.metric" summed over "availability-zone:us-east-1a" grouped by "role".

Here's a screenshot to show what it looks like https://skitch.com/oliveur/emywc/sobotka-datadog

You obviously have a lot of experience with monitoring tools - I'd love to connect. I'm @oliveur on the twitters.

otterley - Don't walk away yet... we do support arbitrary dimensions!

You can attach any arbitrary set of tags to metrics or events - on a per-datapoint basis, and slice / dice / alert based on those tags. Datadog will automatically tag your points by chef role or AWS availability-zone, for example, and you can add any other tag you want. Tags also don't have to be tied to a host and can also relate to a specific volume, mysql index, etc...

(Note to HN, this is a feature of datadoghq - pup will gladly collect and filter on tags, but won't aggregate them, yet)

I should add that Team @datadoghq is on IRC in #datadog on irc.freenode.net

Don't be shy if you have questions when running pup.

We designed pup to be first and foremost accessible to developers, but it will work just the same on production systems.

Once you get addicted to metrics and want more aggregation / graphing / alerting / analysis capabilities, there's a number of open-source components you can pipe your statsd data into. Or you can use our own http://datadoghq.com service for that.

otterley - for that, you can take a look at http://datadoghq.com. We do all of the above.

Our service isn't free beyond 5 hosts, but is quite a bit cheaper than rolling and running your own or flying blind and facing the consquences. It's a hard problem, and we're on a mission to solve it for companies that don't start with goo* or end in *ook.

Thanks katabatic - reducing friction was exactly what we were going for.

Nice work with Graphene, indeed.

In a nutshell, yes. You can also contribute some of the features you need to pup if you are so inclined.

We're on a mission to provide monitoring that doesn't suck, and we believe that making it easy and rewarding to instrument your app is an important step on the way.

I meant it as a shortest time from "I wish I could see my app metrics" to actually seeing them. In other words, ease of set-up, lack of knobs, having metrics appear automatically.

By default, pup aggregates statsd data in 10s intervals, and the UI refreshes it continuously.

This was developed for Datadog, yes. But Pup works fully stand-alone and is completely open-source - so you can take whatever you want / need from it.

Pup has a narrower scope than Graphite (which we also use and love), but optimizes for shortest time to app metrics viz.

(Oh and Datadog itself is all free up to 5 hosts - but remains completely optional)

So: - It doesn't need to run on the same server as the application, and you can have any number of apps reporting to it. The only limitation here is that apps communicate with Pup in UDP.

- Our goal with pup was to make it super-easy to see app metrics. It is therefore much narrower in scope than services like New Relic or projects like Graphite. It's open-source, though, and you can take it anywhere you want.

- We ourselves operate a service that can consume data from Pup and other sources, and provides metrics + events aggregation and correlation, fancy graphing, alerting, etc.. You can check-it out at http://datadoghq.com