My first few weeks at Splunk were very odd. They try to indoctrinate new hires with a barrage of "A-players" that continuously talked about how awesome Splunk was. Except... When I started Splunk was getting their ass kicked by cloud-first players that had recently come to market. Splunk's monolithic architecture wasn't well suited to be run as SaaS at the time and Splunk was burning cash and losing money on every customer that they suckered into moving away from their perpetual licenses into subscription hell. I left money on the table when I ran out the door less than 6 months later.
I'm curious what Splunk's long game is with this because they just told every F2000 that their bottom line is being chipped away by Cribl and friends. So if I'm an enterprising procurement department I'd be tossing Cribl or Rudderstack or whatever other data transformation preprocessor on the table alongside my renewal. Expand opportunity? If you put your ear to the tracks you can almost hear all of the account managers digging out missed quota excuses.
Splunk isn't innovative and hasn't been for a long time. Most of the employees saw the writing on the wall and went to Snowflake as soon as the opportunity presented itself. Splunk tried to capitalize on the security market by, basically, double charging customers for ES. Instead of delivering value it seems to be Splunk is just looking for ways to squeeze a few last drops of lemonade.
Their SaaS offering used to have said inline tier called IDM (Inputs Data Manager) where we were directed to configure filters during our POC… otherwise we never would have moved from Enterprise to SaaS. One month (to the day!) after we moved, they decided to migrate us to a new “Victoria experience” where that tier suddenly disappeared without explanation. We filed support tickets asking 1) wtf happened? and 2) how do we filter things out now? and were directed to hire professional services because that was outside the scope of standard support!
The whole point of moving to SaaS was to not have to babysit our own clusters (small shop at the time), so spinning up a ton of infra in front of the freshly greenlit SaaS setup would have negated the productivity gains and financial pivot.
Ultimately, the entropy of hundreds of applications logging in disparate formats and namespaces outweighed our ability to sanitize each app within a reasonable amount of time, leading to unwanted data being indexed, ergo overages. Overages that our sales engineer originally assured us we could address by filtering things out with the snap of a finger. Bait and switch.
Ingest Actions were not available at the time, and were not functional (even in beta) until 10 months later.
Splunkers have received over 1,020 patents to date
You could have made the same choice, but did not.
If they do discount, even 5%, then it ripples across their accounts as a legal matter, esp at your scale. I was a buyer for some big companies, 8 digit, and the procurement office would only do a deal with MFN/MFC clause. They would also audit the supplier from time to time.
Elasticsearch simply couldn’t handle key collisions. We have hundreds of various apps across 5-10 different languages and frameworks where a key name may be reused as either a string or a hash or an integer or an array. If we can’t freeform search (which Splunk is EXCELLENT at), we just need to be able to transform the data beforehand. Datadog plans to do so with their recent acquisition of Vector.
Instead, each of these systems have their own collectors and correlating from one to the other is hard. A canonical log line is so much more valuable than a metric collected every 60 seconds, and the former can derive the latter: https://stripe.com/blog/canonical-log-lines
Splunk should have been the lynchpin.
Shoving SignalFx down our mouths and trying to get us to create “metric” indices was the straw that broke my back.
I'm sure that's not 100% true but it felt like it. Trying to build Splunk on top of a modern IaC deployment methodology is a huuuuge lift.
Pretty lame not cloud native.
This is the question. If you’re looking for APM well you’ve got great options but for those using Splunk in the security space (SIEM & SOAR) you’re screwed.
There’s no better SIEM alternative that deals with logs at scale.
Splunk recently screwed a friends Fortune 50 company. They didn't pay a bill on time (renewal negotiations) and Splunk without even contacting them just left all the logs from one of their instances on the floor. They lost everything for literally an entire country.
I mean EVERYTHING.
NONE of our other vendors have EVER done that shit in my entire career. EVER. My word means nothing to them; they act like a pure private equity player now.
I think folks that use Splunk for basic search just don't fully comprehend how capable the product is for hunt-type operations when someone fluent in SPL is at the helm.
My frustrations with Splunk have been around their certification and training changes over the years. Used to be able to get a solid tool certificate and decent training materials all for free. It only hurts Splunk though as less people have experience with the tool it lessens their advantage. Makes me disappointed as I really do like the tools itself but literally everything else is terrible. I'd much rather deal with Elastic or go open source with Security Onion.
I don't side with Splunk on their actions but I can't stand customers who withhold payment as a renewal tactic and think that it's equally wrong.
They are mad because Cribl is good at transforming data before it ingested by Splunk, so as to reduce the amount of data that is indexed. Period.
Splunk ONLY RECENTLY released “Ingest Actions” to filter data post-ingest (to avoid indexing) for their SaaS product — something that has always been a mainstay of their on-premise “Enterprise” product. Their ONLY suggestion to filter data that we didn’t care to index in early 2021? Cribl. There’s literally no other reason for us to use Cribl.
I’ve been paying for Splunk since 2008 and can’t wait to get away from them. Their sales teams have decayed into unethical slimebags and I am trying everything in my power to not renew our contracts with them. This just sealed the deal.
Source: I cut checks to Splunk for $x,xxx,xxx yearly