This is a very nice way of framing the question.
I think there may be at least two scenarios to consider. For some products, cost is primarily driven by R&D; for others, cost of operating the service (COGS) dominates. Of course it's a continuum, but I suspect there are more services near the ends than the middle.
For an R&D-dominated service, if your needs are nontrivial and there is an off-the-shelf product that is a decent fit, you probably want to go off-the-shelf. A commercial provider can amortize the R&D over a large customer base.
For a COGS-dominated service, the commercial provider may still be more efficient but it is less of a slam dunk. Log management (e.g. Splunk) involves a substantial operational component, because you're dealing with large volumes of data.
Disclaimer: I am the founder of Scalyr, a Splunk competitor. Early on we realized that, even though log management is COGS-heavy, we could achieve huge economies of scale, because query workloads are highly bursty. By carefully managing a central pool of resources shared by many clients, we're able to run much more efficiently than a single-tenant homegrown solution [1]. This is one of the key points that we highlight when having build vs. buy discussions with potential customers.
[1] https://www.scalyr.com/blog/searching-1tb-sec-systems-engine...
Are there other examples of "we chose to go to market like this (x, y, z reasons) and here's why"?
I am currently using logdna as my splunk alternative, but it's not very good.
Why should I try scalyr next, over the other alternatives ?
My main constraint is the solution should be at least half as good as splunk, and cost less than $50 a month (I generate less than 1G a day).
Your pricing seems to be reasonable. But how do you compare to splunk on features ?
Here are the other tools I am considering trying
You can find a more general overview at https://scalyr.com/product.
Seems like you guys fit the bill. I'll do the 30 day trial, and see how it goes. Thanks for the response.
Steve says elsewhere in the thread that Splunk has the more-flexible query language, and while that may be true I as a product engineer never noticed the delta myself.
TL;DR I think you'd be doing yourself a favor to try Scalyr, and I have no affiliation other than being a happy former customer.
Disclaimer: I'm one of the founders. We built it to be a Splunk alternative and I think it does a great job from a data ingest, scalability, and data querying perspective. We're lacking some of the out-of-the-box capability, but not for long. Kits (our "apps") release is coming this quarter.
If you can answer that, it's all the justification you need for building it. Anything you build that doesn't differentiate you can be bought by a competitor.
There's absolutely nothing wrong with this way of making money; the lower the margins, the more watching your costs is crucially important, and the difference between making a living and not even breaking even!
I just mention it because most people in the world have a job that make them less money than my friends web services.
And I'm assuming most startups, unless heavily founded, may have to think about this trade off as well.
The cost in infra can easily be a factor of ten.
The bigger you are and the more you can think about optimizing spend.
I know for a fact our bill was around 15-20k USD per month for running a webapp that could be ran in digital ocean 40 dollars/month.
For a lifestyle business without VC money to burn it makes all the difference.
Let's say you build some kind of product based on some Azure/AWS product. The negative reading of it is: you spend money on innovation and have to pay Azure/AWS for infrastructure. Your product will constantly have to improve to keep up with the competitors, Azure/AWS keeps getting your money. And if your product becomes stable enough, they will make it part of their offering.
Another example would be Apps, with Apple later folding their functionality into the OS and kicking your app off the store.
But judging by my own examples, this migth be more of a platform problem.
The other side of that coin is that you use a build horizontal integration strategy to replace elements of the products you buy from Azure/AWS if (and only if) it can be done in a way that increase your profit/value.
It can work both ways. Moving from buy to build also has elements of de-risking and economies of scale if you do it at the point where there is measurable success/product to market fit.
Building something you can buy too early runs the risk of premature optimisation if not done for the right reasons.
Though it sets up an interesting situation for me, where the "special sauce" of _my product_ is the 80% "off the shelf" for my customers.
By contrast, whenever I've been on a project where the company went straight to building their own, it ended up being a quagmire of scope creep, and the development team would end up stuck on an endless treadmill of implementing features that stakeholders vehemently insist on, but never actually use.
Even for dev tooling and libraries. Some of the worst piles of technical debt I've encountered happened when some rockstar decided that they knew better than everyone else, and dived straight into building what they thought would be a better mouse trap. It's not that the in-house option isn't any more or less likely to solve the problem well (no comment), so much as that the in-house option is going to end up being more deeply tangled with the rest of the codebase. So you're stuck with it, because, even if it doesn't turn out to be all it's cracked up to be, migrating away from it may be next to impossible. Homegrown ORMs seem to always turn out this way.
The one exception is when you know that a quick-and-dirty DIY solution will be cheaper and easier, because it needs to do only one specific thing while the off-the-shelf option has to take on a bunch of extra complexity in order to be general enough for everyone's needs. But even there, I'm only likely to trust that argument if it's being made by someone who's been burned by the complexities of the off-the-shelf solution in the past, and who has a proven track record of obeying the KISS principle.
Brooks' software system products (9x the work) is usually done this way in practice. In most usage cases, I would also include open source projects.
It's so hard to get interfaces right and it's helpful to have many users. But I considered your point, of it helping to enforce boundaries.
Their pricing is ridiculous and what is even stranger is that with the ELK stack there is a good and free alternative. Now Splunk is good, no doubt, but I still wonder how they can be that successful with that pricing.
So people start running anything — from dashboards to analytics — from Splunk alone. Their ability to combine and extract data from almost anything in very small amounts of time is unparalleled IMO.
What if there are significant costs associated with that? Integration, training, friction to move to a new solution later, etc?
When things are commoditized it works OK but for anything complex and strategic, the cost of implementing a potentially bad solution can be absolutely crippling, especially if dependencies on that solution grow rapidly within the organisation.
Part of this story is here https://netflixtechblog.com/introducing-winston-event-driven...
Disclamer: StackStorm founder here (but not pitching).
As an example, we have a pipeline of services. I can compute the time spent in each service with multiple levels of percentiles and group that data by high cardinality fields (as in, hundreds of thousands or more values). I just did a search for 4 hours of data across thousands of nodes for half a dozen or so services with multiple eval statements all piped to a timechart doing over a dozen stats operations. Half a billion events. It got done in under a minute.
Splunk charges so much because they are just so dang powerful.
In the early days Splunk pricing was exorbitant (we evaluated Splunk 7 years ago and dismissed it), but licensing has changed in recent years and it is now priced by volume ingested (the pricing is transparent and listed on their website now). At low volumes, the pricing is similar to Sumologic, and is pretty accessible now to smaller dev shops. Open-source collectors like fluentd also help to intelligently reduce the ingest volume.
At high volumes, the TCO changes quite a bit.
Their profit margin is usually in their efficiency in doing it better than you, and at the same time you get the benefit of all the features they add for other customers.
Oftentimes the decision to build usually came down to the difficulty in integration of a commercial product, not the cost. In most cases the cost is set to be competitive against building yourself.
Except in the case of Splunk. Their pricing is ridiculous.