Developers being absolutely terrible at pricing is not unintuitive (I'm a developer)
Great product, but completely useless utility value with financial considerations for environments with high volume.
Wow, it's THAT expensive?
There's even reluctance to turning things on and _watching_ it for 10 minutes. An activity that would immediately give you a much better idea of volume. Folks just don't like doing it.
Then you get the things were setting up a redundant logsource is just unwise. DNS logging was 2 orders of magnitude greater than everything else a SIEM was doing. And Email was about the same size.
(If you aren't careful and aren't managing your costs, but I suppose that's true of almost anything =)
Splunk was used by a much larger product (easily 10x our scale) for monitoring events so there was no red tape to start using it.
After launching the detailed instrumentation (1 structured log event per HTTP request with a breakout of database/service activity) I was able to gain all of the insight needed and build a simple user/url lookup dashboard page to help other engineers see what was going on. We went from being mostly blind to almost full visibility in less than two weeks.
The downside was, we increased our billable Splunk usage by 50% since we were capturing so much more data per log event than the other product just consuming standard IIS/Apache logs.
That type of flexibility was totally worth it. Due to some acquisition shenanigans we broke off from that group and wound up on ELK stack which didn't perform quite as well, but was still usable with the same data. In today's day and age we could have just built an OpenTelemtry library.
Another solution is pre-processing (serial dupes are not forwarded).
Another solution is heavily reduced logging (ERR or higher only on prod hosts).
These can be used together and be very helpful.
With that said Splunk should offer such a pre-processing product (maybe it does?) which would probably increase their moat even though it reduces revenue somewhat in the near term.
https://www.splunk.com/en_us/blog/platform/introducing-edge-...
https://www.splunk.com/en_us/blog/conf-splunklive/ingest-act...
So it doesn’t need to be expensive, naturally, it just is.
Back in the day, I worked for an automation company. The software was called "Blade Runner." This was in the era when blade servers were all the rage.
Turns out you can't use a copyrighted name, so they changed it from "Blade Runner" to "BladeLogic." (All the directories are still "BR" btw.)
A bunch of the people from BladeLogic pulled a Mark Zuckerberg and moved from the Boston area to the Bay Area and founded SumoLogic, hence the name.
Oracle isn’t just expensive, it also has to be technically horrible but still operational.
fail to monetize the light?
It seems to me the marriage between APM and logging would be a home run.
* https://www.splunk.com/en_us/about-splunk/acquisitions/signa...
* https://www.splunk.com/en_us/blog/conf-splunklive/introducin...
Someone (not to name names) got bitten by the "anti-weirdware" bug and started shifting us off of all our custom-built solutions. Every team got hit with major distractions from their roadmaps for each of these changes. None of the headcount dedicated to staffing the internal systems was freed up - they had to run the new integrations.
The decision was made one day to migrate all of our observability stuff over to SignalFx. Observability wasn't our "core competency" and our systems were "weirdware".
We had to rewrite our instrumentation, all of our reporting dashboards, and all of our alerting DSLs changed. They were not replaced 1:1 for every system and metric, so we emerged in a much worse, much less visible situation across the board. Outages happened or went unreported.
Splunk acquired SignalFx and dramatically raised prices. We scrambled to do the migration process yet again, impacting roadmaps and leading to more outages.
Leadership was changed.
There's one thing to be said about NIH, but when you write systems that are already working, inexpensive, and easy to maintain, you shouldn't throw them out because you're worried analytics isn't your "core competency". Yes - it is your core competency, because you're selling uptime to your customers.
Any lower priced alternatives? Or self-hosted?
Splunk / Datadog have the classic user interface lead of a closed source solution, but IMO that premium's days are numbered.
I’m always happy when I can use some of our sources that are in splunk but get sad that I can’t do that with everything else.
Its cloud pricing is funny because it’s so much more powerful with massive amounts of data, but they charge based on storage. Our on prem instance wasn’t just simpler to price but we could throttle resources to allow for really high volumes of data with relatively slow query and analysis.
Anecdote, I once worked with a client that really wanted to get Splunk, but produced so much network traffic that the discounted annual costs were more than the entire budget for the rest of the organization combined. That's staff, the building, equipment, power, water, everything...the estimated Splunk cost was more than that.
They went with a combination of ELK and a small team of dedicated developers writing automation and analytics against Spark and some enterprise SQL database. Still expensive, still cheaper than Splunk.