Re:What I Learned Managing Site Reliability for Some of the ...
by user5994461 on Tuesday April 04, 2017 @08:20PM
(#14038570)
Splunking is cool.
Nonetheless, splunk is the most expensive software license on the planet. More than Oracle, yes.
Re:What I Learned Managing Site Reliability for Some of the ...
by jsmthrowaway on Tuesday April 04, 2017 @08:24PM
(#14038589)
I meant spelunking, where they got their name. But yes, you're not wrong. (I've heard of $millions.)
More than once I've run the entire corpus of requests to a system, ever, through a dummy rebuild as a pretty great integration test. It's a powerful SRE tool. Spelunking through all historical data is just icing on that cake, honestly. As the author says, SRE is basically just an information factory; I'd be haaaaaard pressed to agree with you on throwing away a lot of information -- you don't know what you don't know until you want to know it -- and betting all-in on monitoring. Retaining logs is not the hardest problem SRE deals with, either, but SREs turn around and force unrealistic latency requirements on the query side (I see a lot of ELK deploys running into this).
You have to look at it as an Oracle. Oh, great Oracle of a pile of meaningless logs, cook off this map/reduce and tell me an interesting number that I can put in a Keynote for executives. Definitely not dashboarding from logs data. That's an impedance mismatch that Google gets away with because of the nature of their logging.