It's not the log storage that we care about, it's what we can do with those logs.
I've been through three logging vendors in the last two years (Loggly, LogEntries, now Sumo Logic) and I'm looking to move once again. The secret sauce is not the storage volume; our volume is relatively small: 4.3 GB/day. The secret sauce is the intelligent structuring of unstructured log data, analytics+alerting on the now-structured data, and live tail'ing of logs.
Sumo Logic does an okay job at the parsing/querying/alerting/analytics piece but live tailing is a brand-new feature for them and it has limitations. Sumo Logic's pricing is also frustrating--they don't apply volume discounting until you're pushing something like 500 GB of logs each month. Again, we need affordable analytics; storage is a small piece of the puzzle.
Honestly, our next move may be to an internal "HEK" stack: Heka, Elasticsearch, and Kibana. I've looked at the cost of doing this and it's actually more expensive when you consider what we'd spend to host a big ES cluster internally, but cost is not our #1 consideration, usability is.
Congratulations to the LogDNA team on the launch. There are a lot of use cases in this field and it could use some more shaking up.
(Disclaimer: I'm the founder of Scalyr.)
Scalyr looks interesting, how does it compare to Sumo Logic feature-wise? For reference, this video has made me switch to them: https://www.youtube.com/watch?v=w5yWfYDQz2Q
Speed turns out to be surprisingly important. We've had a lot of customers report that after migrating, usage across the team goes way up, simply because it's much more satisfying to use a tool that gives quick results. See https://www.scalyr.com/case-studies/returnpath for a brief case study.
I'd be happy to discuss in more detail, but I don't want to hijack the thread - please drop me a line at [email protected].
http://www.splunk.com/en_us/products/splunk-enterprise/featu...
Give us a try, even if you don't think you'll use the GBs. We have alerting, field parsing and live tail. Analytics is the only piece we don't have today, but we're working towards it.
The number one most important problem we're trying to solve is the actual parsing of logs during ingestion. We need logs from commonly-used apps & frameworks to be automatically identified and parsed and we need an easy way to set up a custom parser for internal logs that doesn't require crazy hand-written regex. How does LogDNA handle parsing? What apps/frameworks are supported?
We run our ELK stack (elastic search+logstash+kibana+3 days of logs) on a single EC2 ephemeral instance inside an auto-scaling group. If we lose our logs, oh well! Given that we lose this instance every two-ish years, it seems like a reasonable tradeoff.
Have you tried that? I wonder how Mozilla likes it
To be clear, I'm not asking that you have all of this out of the gate. It's that I want to see the vision and get excited about where you're heading. Your early users are investing in your future and taking a bet on what you could become.
Right now, I'm not sure what I should be dreaming about.
Obviously internally (as a fellow founder), things are always crazy and you adapt and grow. But your customers need more than the faith that you'll figure it out as you go.
I was referring to the aggregation of the logs, including taking columns / data types from different log streams and combining them (in real time) to create new objects. You can then set alerts and build dashboards on the new objects.
Additionally he mentioned standardizing the request across the stack. Since Striim can import the logs from all these types of services and combine them to create a new object that represents the request across the stack he could accomplish his second request as well.
Combine that with the ability to ingest huge amounts of data and query in real time, it sounded like what he was looking for.
What's the killer use case or differentiator (vs. loggly+kibana+elasticsearch for example)? With Slack, it is integrations. If you could do something similar, like a bugsnag + new relic that aggregates all of our logs and notifies us when things happen (either individual incidents or aggregates like "nginx error log above 1 event/10s), you can have all my money.
Another idea would be to standardize transactions across the stack, so I could trace a request from nginx through rails/phoenix, to postgresql, etc. Again, take my money.
A better UI only helps me if I know when to look at it or what I'm looking for.