Not so - Splunk has a very sophisticated index for huge amounts of data, and a rather fancy DSL for analyzing it. I've looked at it in some depth as one of our products (http://zetetic.net/software-combine) does log aggregation (and does a much better job with Windows and SNMP than does Splunk), but Splunk is not at all a trivial piece of software.

When you said DSL I head Regex.

Honestly, your competing with inhouse scripts people spend a few weeks / year maintaining. Which is not to say you can't make a lot of money doing so. Ideally, your saving people significant time writing these scripts, but tossing out buzzwords just alienates your users.

Splunk's query language is not like regex - it's somewhat more like SQL or LINQ, with groupings, aggregates, pivots, etc. It is definitely a DSL.

I am not a Splunk developer or serious user of it, but you greatly underestimate the problem space to suggest this class of products address the same problems solved by even a very skilled scripter's scripts.

It's much more than that. It lest you query logs, and queries can be very complex, which then gives you a real time result with charts and all. I've used it in production, and it's was the best damn tool to debug production problems. The licensing costs are insane though.

What are the licensing costs?

It really depends on the size of the logs, the number of users, and the deal you can negotiate. We paid around $60K/year, but we had quite a lot of logs.

syslogd + open source monitoring + indexing has done this for a while now.

Competing business idea...?

As a guy who used to resell Splunk, it offers a lot more in comparison to it's open source alternatives. Being data agnostic with decent log compression, with an extremely rich searching syntax, some very good (HTML 5) reporting dashboards, decent reporting and a verbose API make it very attractive for users (companies) who have compliance requirements or require some form of performance monitoring. (I know it's a mega sentence)

I have since left working with Splunk directly but I would still advocate its use because it's one of the better commercial (albeit expensive) log management/SIEM products around.

Log data is:

  * the largest data source in any enterprise
  * increasingly must be stored for long-term analysis ...
    * by corporate policy or ...
    * by government regulation/demand

Regular open-source tool stacks can't do this without astronomical cost in storage hardware. Various log analysis players have managed to make this data storable and some have done well on the analysis side.

Splunk will do well. I hope this boosts acquisition interest in my old company, http://www.sensage.com, another log analysis company.