Re:Splunk (big data) IPO Raises $229.5 million ($1.6 billion...
by invertd on Thursday April 19, 2012 @06:15PM
(#3865432)
Looking through their site it looks like a Splunk search (or a query) can include a large set of non-trivial, CPU intensive operations. Therefore, perhaps, it is a process that does not lending itself well to multithreading: http://splunk-base.splunk.com/answers/12027/singlemulti-thre...
Re:Splunk (big data) IPO Raises $229.5 million ($1.6 billion...
by nl on Thursday April 19, 2012 @07:38PM
(#3865767)
That's true as far as it goes.
Nevertheless, I believe that there are opportunities for query multithreading that aren't being taken.
For example, a query like this appends the second query results to the first, and the graphs both:
sourcetype="blah" | search blahblah | eval series="label1" | append maxtime=600 [search anotherlongsearch | eval series="label2" ] | timechart count(somefield) by series
There is no reason why that second search couldn't be executed simultaneously, and that would approximately half the time for the whole query to run (assuming sufficient CPU power etc).
It's pretty good software, and isn't as trivial as some on this thread seem to think (I've built large Solr implementations too, so I know search reasonably well).
The strong points: good interface, excellent data import, decent search language, decent docs & community, good APIs, a good set of mostly decent drop in applications that run on top of it.
The weaknesses: While indexeing is Map/Reduced based and scales fairly well, querying is single threaded. That limits it to the performance of a single CPU core + IO limitations. This also applies to things like sub-queries: in a database they could be run separatly, but in Splunk they aren't.
It is also fairly expensive at large scale, although the licening model is fair (it is licensed by data volume, so you can install it on as many machines as you like and share the license between them).