News for hackers. Stuff that matters.
Re:Datadog's $65M/year customer mystery solved by ljm on Monday June 30, 2025 @04:24PM (#44427444)
I wonder how much that no-expense-spared, money-is-no-object attitude to buying SaaS impacts an engineers ability to make sensible decisions around infra and architecture. Coinbase might have been fine blowing 65 mil but take that approach to a new startup and you could trivially eat up a significant amount of runway with it.

I won’t single out Datadog on this because the exact same thing happens with cloud spend, and it’s very literally burning money.

Re:Datadog's $65M/year customer mystery solved by viccis on Monday June 30, 2025 @05:44PM (#44428240)
>I wonder how much that no-expense-spared, money-is-no-object attitude to buying SaaS impacts an engineers ability to make sensible decisions around infra and architecture

I saw this a lot at a previous company. Being able to just "have more Lambdas scale up to handle it" got some very mediocre engineers past challenges they encountered. But it did so at the cost of wasting VAST amounts of money and saddling themselves with tech debt that completely hobbled the company's ability to scale.

It was very frustrating to be too junior to be able to change minds. Even basic things like "I know it worked for you with old on-prem NFS designs but we shouldn't be storing our data in 100kb files in S3 and firing off thousands of Lambda invocations to process workloads, we should be storing it in 100mb files and using industry leading ETL frameworks on it". They were old school guys who hadn't adjusted to best practices for object storage and modern large scale data loads (this was a 1M event per second system) and so the company never really succeeded despite thousands of customers and loads of revenue.

I consider cost consideration and profiling to be an essential skill that any engineer working in cloud style environments should have, but it's especially important that a staff engineer or person in a similar position have this skill set and be ready to grill people who come up with wasteful solutions.

Re:Datadog's $65M/year customer mystery solved by nasmorn on Tuesday July 01, 2025 @05:20AM (#44432069)
It is also not a very hard skill. You do a back of the envelope calculation and if your proposed architecture is crazy expensive for your reasonable load, then you have to figure out if you are a special snowflake or just doing it wrong.
Re:Datadog's $65M/year customer mystery solved by viccis on Tuesday July 01, 2025 @05:12PM (#44438048)
This is correct. It's really more of a mindset than anything. You take a guess at how much something will cost based on a quick calculation (good cloud providers make this easy, some cough Databricks cough just use a black box and bill you whatever they feel like) and then once you test it at a small scale, you verify that it's as expected and continue to monitor.
Re:Datadog's $65M/year customer mystery solved by happymellon on Tuesday July 01, 2025 @05:32AM (#44432131)
What's also frustrating is that a lot of times, costs are hidden from engineering.

I don't know if I would call them mediocre, but without a feedback loop its hard to get engineers to agree whether it's worth time reviewing the code to make it faster compared to just making the db one size larger.

Re:Datadog's $65M/year customer mystery solved by viccis on Tuesday July 01, 2025 @05:20PM (#44438105)
>costs are hidden from engineering

Yeah one of my big pet peeves was when engineering teams build platforms to run things on that obscure the cost. There have been times where they said "hey we made this big platform for analytics, just ship your stuff as configuration changes and it's deployed!" Then when I did it with very simple small cases, some unoptimized stuff on their end (a lot of what I talked about before) resulted in runaway costs that they, of course, tagged to my team.

Ultimately, you can only control what's in your scope and anything else you will need to hope that management can take that runaway cost feedback and make the correct team optimize it away.

>I don't know if I would call them mediocre, but without a feedback loop its hard to get engineers to agree whether it's worth time reviewing the code to make it faster compared to just making the db one size larger.

This started in the mid 2010s, by which point they should understand that you don't put terabytes of data into S3 in 100kb files. And if not, they should be willing to take some very simple steps to address it (literally just bundling them all in 100mb files with an index file containing the byte offsets of the individual ones would have solved a lot of their problems). There was a feedback loop. There just happened to be big egos more interested in their next fun project of reinventing another solution to another solved problem. I learned there that engineering driven companies sometimes wind up in situations in which the staff engineers love fun new database and infrastructure projects like that more than they enjoy improving their existing product.

Re:Datadog's $65M/year customer mystery solved by happymellon on Tuesday July 01, 2025 @05:55PM (#44438335)
Yeah, that sounds terrible.
Re:Datadog's $65M/year customer mystery solved by closeparen on Monday June 30, 2025 @06:37PM (#44428683)
That's the point of usage-based pricing: it's cheap to adopt when you're small.
Re:Datadog's $65M/year customer mystery solved by swyx on Monday June 30, 2025 @04:44PM (#44427650)
the visible cost of burning runway on a bill is very often far less than the invisible cost of burning engineer time rebuilding undifferentiated heavy lifting rather than working on product/customer needs
Re:Datadog's $65M/year customer mystery solved by pphysch on Monday June 30, 2025 @04:57PM (#44427799)
Most of the complexity in observability is clientside.

It is not hard to spin up Grafana and VictoriaMetrics (and now VictoriaLogs) and keep them running. It is not hard to build a Grafana dashboard that correlates data across both metrics and logs sources, and alerting functionality is pretty good now.

The "heavy lift" is instrumenting your applications and infrastructure to provide valuable metrics and logs without exceeding a performance budget. I'm skeptical that Datadog actually does much of that heavy-lifting and that they are actually worth the money. You can probably save 10x with same/better outcomes by paying for managed Grafana + managed DBs and a couple FTEs as observability experts.

Re:Datadog's $65M/year customer mystery solved by lerchmo on Monday June 30, 2025 @05:22PM (#44428025)
You could hire 100 people to manage your timeseries data and save 70%
Re:Datadog's $65M/year customer mystery solved by ljm on Tuesday July 01, 2025 @09:41AM (#44433840)
I used to be quite fond of Datadog but after one or two completely surprising bills (thanks to their granular but unintuitive pricing model), I wouldn't recommend it to anybody any more. If I were more cynical I would say the pricing model is designed to be confusing so customers spend more than they need, and this is only made worse by the extreme breadth of the platform now.

These days I'd suggest to just suck it up, spin up a Grafana box, and wire up OpenTelemetry.

Re:Datadog's $65M/year customer mystery solved by 9283409232 on Monday June 30, 2025 @04:52PM (#44427744)
People say this but I wonder about this from time to time. I don't think anyone is asking to rebuild datadog from scratch for your company but surely it's worth it to migrate to something not as expensive even if it takes a bit of elbow grease.
Re:Datadog's $65M/year customer mystery solved by closeparen on Monday June 30, 2025 @06:38PM (#44428690)
Assuming there's nothing else you could do with that elbow grease that would create more value than the SaaS bill costs.
Re:Datadog's $65M/year customer mystery solved by 9283409232 on Monday June 30, 2025 @09:46PM (#44429796)
Value is not a hard science. I've seen people shelve tech debt in favor of work on a feature that no one ends up using.
Re:Datadog's $65M/year customer mystery solved by nemothekid on Monday June 30, 2025 @09:54PM (#44429841)
1. Leadership doesn’t want to burn engineer cycles on undifferentiated features.

2. Management doesn’t get recognized for working on undifferentiated features.

3. Engineers working on undifferentiated features aren’t recognized when looking for new jobs.

Saving money “makes” sense but getting people to actually prioritize it is hard.

Re:Datadog's $65M/year customer mystery solved by uaas on Tuesday July 01, 2025 @02:26AM (#44431152)
Well, saving money is a differentiator, and one of the best things an engineer can put on their CVs.
Re:Datadog's $65M/year customer mystery solved by QuinnyPig on Monday June 30, 2025 @09:56PM (#44429854)
This is very well stated.
Re:Datadog's $65M/year customer mystery solved by swyx on Tuesday July 01, 2025 @01:49PM (#44436361)
haha thanks Corey, i echo the best (you)
Re:Datadog's $65M/year customer mystery solved by wavemode on Monday June 30, 2025 @11:45PM (#44430384)
I wouldn't really say "very often". Occasionally, perhaps.

Even from a pure zero-sum mathematical perspective, it can make sense to invest even as much as 2 or 3 months of engineer time on cloud cost savings measures. If the engineer is making $200K, that's a $30000 - $50000 investment. When you see the eye-watering cloud bills many startups have, you would realize that, that investment is peanuts in comparison to the potential savings over the next several years.

And then you also have to keep in mind that, these things are usually not actually zero-sum. The engineer could be new, and working on the efficiency project helps them onboard to your stack. It could be the case that customers are complaining (or could start complaining in the future) about how slow your product is, so you actually improve the product by improving the infrastructure. Or it could just be the very common case that there isn't actually a higher-value thing for that engineer to be working on at that time.

Re:Datadog's $65M/year customer mystery solved by happymellon on Tuesday July 01, 2025 @05:34AM (#44432140)
> It could be the case that customers are complaining (or could start complaining in the future) about how slow your product is

If Jira has taught me anything, it's that ignoring customers when they complain its too slow makes financial sense.

Re:Datadog's $65M/year customer mystery solved by JohnMakin on Monday June 30, 2025 @06:19PM (#44428533)
> Coinbase might have been fine blowing 65 mil but take that approach to a new startup and you could trivially eat up a significant amount of runway with it.

Most startups are not going to have anywhere near the scale to generate anything approaching this bill.

> I won’t single out Datadog on this because the exact same thing happens with cloud spend, and it’s very literally burning money.

Unless you're in the business of deploying and maintaining production-ready datacenters at scale, it very literally isn't.