It's around 6 data sources on ~25 machines, but it could be easily scaled to way more than that with a bit of work. And I mean less work than it takes to do even trivially simple things using the horrible Splunk API. There are many thousands of small companies using Splunk and getting totally ripped off for a very mediocre product with a rapacious and annoyingly aggressive salesforce.

I think we're talking about very different levels of scale. Enterprises are generally feeding tens to hundreds of thousands of datapoints into Splunk depending on their size between servers, networking gear, endpoint devices, etc.

That is a tiny setup all things considered. You aren’t operating at a scale you’d need to consider a monitoring platform for.

You'd be surprised how many companies with infra that small have CTOs get consultant buzzword pilled into buying every SaaS under the sun nonetheless...

How many servers does Stack overflow run on? It’s not a good measure of data volume or criticality.

I think “expensive” here is basically relative to revenue/margin. Where margins are high, spending on Splunk (etc.) isn’t meaningful. Where margins are thin, it hurts.

Basically, the arguments here seem to reflect the markets and business model folks are working under. Some pay, some can’t and some won’t - all valid.

But you definitely want to, even if it simple ELK stack

I have an order of magnitude more machines than you and would never in a million years consider splunk

Right tool for the right job. Splunk is for mega-scale setups

> it could be easily scaled to way more than that with a bit of work.

I guess you'd appreciate the words easily and bit are doing a lot of heavy lifting there.

> I guess you'd appreciate the words easily and bit are doing a lot of heavy lifting there.

This goes with the previous comment:

> And oh yeah, it's totally free instead of costing my company thousands of dollars a year

Unless you work for free, then something you make and maintain is not "totally free".

Liiiiissssteeeennnnn

I havent developed it yet. But my Splunk killer solutions actually scales so big we can use it to walk to the center of the universe. And its only 1 line of Rust and a bash script that runs when ever the Unix clock has 420 in the number string.

ya as someone else already noted - Splunk is not for you

Splunk isn't perfect. Managing it is more work than it should be for example. But I've got hundreds of systems I'm pulling logs from and that's not counting infra and applications as well. And my deployment isn't even a large one by their standards. Your use case just isn't the scale where splunk makes sense.

Wait what this is such an important detail. Log aggregators like Splunk start being something to consider when you get to about 25 THOUSAND machines, not 25 machines. I hope that for you, humility will come with experience.

Splunk does not scale to large data sources. It fucks out at a few TB and then you have to spend hours on the phone trying to work out which combination of licenses and sales reps you need to get going again.

By which time you can just suck the damn log file and grep it on the box.

I'm gonna respectfully disagree that it fails "at a few TB". We send them 100s of terabytes a day.

But, and this is not meant as criticism or insult as I have no idea how Splunk works, it is just based on other comments; do you know what license your company has with them? It appears that if you are paying them millions, it scales fine, otherwise, it does not?

> I have no idea how Splunk works Cool

> It appears that if you are paying them millions, it scales fine

yes, if you pay someone for product and services, you get them. If you don't, you don't

It's difficult to control data ingress so you end up in debt and on repayment plans. Which are expensive.

That makes sense, so looking at what people ingress, they pay afterwards or just really huge plans upfront? Or a mix?

Well usually you have to overpurchase up front and they sell you a 3 year lock in to make it affordable capital cost. Then when you eek over it temporarily, the sales guy calls you up within 10 nanoseconds to bill you for more.

I was getting 2-4 calls a week.

It was so fucking annoying and expensive ($1.2M spend each cycle) we shitcanned the entire platform.

First thing they hear of this is when our ingress rate drops to zero and they phone us up to ask what is happening. Then we don't go to the numerous catch up and renewal meetings and calls. Then we stop answering the phone.

Had a similar experience with them, they are truly the worst. We wasted a bunch of time trying to figure out how the ingestion volume could be so high and then realized that 99% of it was from the ridiculous default settings of their universal collector agent which was dumping detailed system stats every few seconds-- all to drive up usage so they can harass you about spending more money on their awful product. I did the renewal call with them just to basically tell them how outrageous their company is.

Yeah, because that is what I meant. A lot of services are useable without paying through the nose, this one apparently not, but thanks for the excellent input.

I'm certainly not a Splunk expert and I'm CERTAINLY have no insight into the nature of our financial arrangement with them, but yeah it's expensive.

I think there's not much of a useful "flat rate" tier; you pay based on usage. People can accidentally spin up a ton of EC2 instances and get a huge surprise AWS bill, too. And yeah our logging needs are high and monotonically increasing but they're also relatively predictable at our scale.

It ALSO turns out though that Splunk is really really good at their job and matching their expertise would require tons of engineering effort and it's not like the disk space alone is THAT cheap if you want it to be searchable.

I've worked at companies with objectively large amounts of data. Splunk scaled to meet their workloads. At no enterprise doing this is someone able to just isolate a single log file and grep through it at scale.

Presumably you can have a cluster of grepping machines. I wonder how it scales compared to the millions you pay for Splunk.

is your business' core competency building a distributed grep or actually selling useful stuff?

Well, according to what people write in this thread, a distributed grep or some other way to organize a decent central logging system might be a necessary part of the core competency. Because if they buy splunk instead, they might go bankrupt.

You don’t have to be splunk to make money out of distributed grep but it turns out to not be that easy… as proven by the fact that there are quite a few competitors

My core competency is saving the business tons of money, so that they can pay me.

Uhhhh you splunk scales no matter the size. for just pure ingest. Now if you got duped into the SVC model I can see what you mean. But for pure Gigs/Day ingest if you know what youre doing it can scale infinitely.

I remember a client using Graylog. It was good for app logging and is available as open source.