News for hackers. Stuff that matters.
Re:DataDog asked OpenTelemetry contributor to kill pull request by hosh on Thursday January 26, 2023 @07:21PM (#34540812)
We use DD, and we have to use Otel for our Elixir apps to ingest spans into DD because there are no DD SDKs for Elixir or Erlang.

From the perspective of a customer, I can tell you that DD already has quite a bit of a moat. Their main competitive advantage, and what got us into using it, is being able to correlate data across APM, custom metrics, and logging through the use of tagging. They then densely link data together across the platform. There is also a built-in Jupyter-style notebooks. By correlating data like that, you get more value out of ingesting as much data into DD as you can. There are some additional services we're not using, such as auto-correlation with ML (and alerting for anomaly detection), and security monitoring that also looks across the entire platform using their ML tech.

Like AWS/GCP/Azure, it can get expensive, quite fast, using on-demand pricing, so there are negotiated annual contracts. Right now, our team is small, and to replicate the functionality we do use, using self-hosted open-source tooling, we might as well hire another engineer for just setting up and maintaining such a platform.

I get it that, you want to defend the moat and that eroding the little things can lead to eroding the big things. As I see it though, if you need those correlations, you'll need a certain scale and team size before it makes sense to build out something like that for yourself.

Re:DataDog asked OpenTelemetry contributor to kill pull request by aeyes on Thursday January 26, 2023 @07:37PM (#34540967)
> negotiated annual contract

Maybe we are too small but Datadog is one of the few vendors which we haven't been able to negotiate down in years. The price has always been whats on the website. I honestly don't even mind, with some vendors it feels like you are on a basar and they always tell you that their final discountns had to get approval by the CEO.

Re:DataDog asked OpenTelemetry contributor to kill pull request by hosh on Thursday January 26, 2023 @07:42PM (#34541013)
I don't know if we got a discount for an annual contract, or if the annual contract pricing was published on the website. But we were able to work out things like, using seasonal pricing because our traffic is seasonal.
Re:DataDog asked OpenTelemetry contributor to kill pull request by slyall on Thursday January 26, 2023 @08:15PM (#34541258)
I always tell people looking at Datadog to remember "DDDD" or "DataDog Don't Discount".

Compared to most Enterprise vendors it is a lot harder to get a discount from Datadog. Most vendors will give you 1/3 off just for signing a contract and committing to a spend, Datadog is not like that.

Re:DataDog asked OpenTelemetry contributor to kill pull request by shyn3 on Thursday January 26, 2023 @10:17PM (#34542043)
Just show them New Relics new pricing
Re:DataDog asked OpenTelemetry contributor to kill pull request by wink on Friday January 27, 2023 @03:04AM (#34543761)
the New Relic who either had a breach they never notified me or sold my email address? I've getting random spam to newrelic@<domain> for years now. Nope, they'll never see any money if I can avoid it.
Re:DataDog asked OpenTelemetry contributor to kill pull request by Hawxy on Friday January 27, 2023 @03:11AM (#34543816)
> Maybe we are too small but Datadog is one of the few vendors which we haven't been able to negotiate down in years.

We spend a few thousand a month with Datadog and our account manager reaches out every quarter to adjust our monthly commit up/down which provides a 20% discount (I think) or so off from the website prices.

Re:DataDog asked OpenTelemetry contributor to kill pull request by ChrisCooneyUK on Friday January 27, 2023 @05:05AM (#34544418)
Full transparency, I work for Coralogix, but customers moving from DataDog to Coralogix regularly see a 40-70% cost reduction. And we actively encourage our customers to build using OpenTelemetry because we have confidence in what the product brings :)

https://coralogix.com/docs/opentelemetry-using-kubernetes/

Re:DataDog asked OpenTelemetry contributor to kill pull request by pranay01 on Thursday January 26, 2023 @07:43PM (#34541025)
you may want to check out https://github.com/SigNoz/signoz

I am one of the maintainers. We are building a DataDog alternative with native support for opentelemetry.

Re:DataDog asked OpenTelemetry contributor to kill pull request by hosh on Thursday January 26, 2023 @07:56PM (#34541130)
I have seen it around, but didn't know it was open source. If you guys are working on correlation, then I will definitely be watching this. Consistent link urls that can be shared is also useful.

Hopefully someone else will contribute the notebooks feature. Those are very useful.

Something that DD is not careful about, is being able to consistently use UTC for all time labels in all graphs (and maybe a quick way to convert to a local time if we need to communicate with stakeholders).

(I don't know why your comment was downvoted).

Re:DataDog asked OpenTelemetry contributor to kill pull request by pranay01 on Thursday January 26, 2023 @08:11PM (#34541223)
Yeah, we are actively working on correlation. We have metrics, traces and logs in a single app - so we should be able to provide a seamless correlation.

Thanks for the point about Notebooks, we have not thought in detail on how people use that. Is it primarily to collaborate between team members when an incident happens or even when there is no incident, and you are analysing stuff

Re:DataDog asked OpenTelemetry contributor to kill pull request by hosh on Friday January 27, 2023 @11:29AM (#34547874)
We have used those notebooks for:

- Incidents, collecting different metrics and showing them next to each other, with comments

- Longer-term reliability debugging. They can form a kind of ad-hoc dashboard. These are usually issues that degrade performance, don't have immediate or wide-spread customer impact, and are things we are not immediately able to detect

- Related, performance tuning. Sometimes, the key metric is unknown. We want to explore it, and then make changes to infra, and then see if that moved the needle

- Sometimes, the ad-hoc widgets are useful enough to export to a dashboard

- I can take any widget anywhere else and import it into a notebook, or start a new notebook out of it.

The notebooks are similar to the dashboard, just that, the layout engine only allows a linear notebook layout instead of a grid. There are already text widgets, though the button to add that is easier to access. Other than the comments, it's basically a dashboard with the UI changed so that it feels like a notebook.

Keep in mind too, all dashboard and notebooks modify timestamps and other states in the browser URL, so it is easy for me to copy-paste those into Slack so that other people can see what I am seeing.

Re:DataDog asked OpenTelemetry contributor to kill pull request by SOLAR_FIELDS on Thursday January 26, 2023 @10:04PM (#34541968)
We use Datadog at the medium sized unicorn I work at. People often don’t understand how important correlating timestamps is. I wish Datadog was just opinionated as hell and said everything you see is UTC by default. Or even less dramatic than that, literally just slap the time zone on whatever it is that you’re viewing. It’s so insanely critical when you’re viewing logs on a prod outage that people immediately see and understand that the logs they are looking at are PST, CST, UTC or whatever. It’s insane that software designed to help people unfuck production systems doesn’t display that by default.

As-is we go through a song and dance whenever we look at logs and metrics “oh, this happened at X time which is Y time for most people.

Re:DataDog asked OpenTelemetry contributor to kill pull request by hosh on Friday January 27, 2023 @11:31AM (#34547909)
Our ops team standardized on UTC (though not all of our tools support that).

When we talk to stakeholders and customer-facing folks though, tend to convert it to local time.

Re:DataDog asked OpenTelemetry contributor to kill pull request by harpratap on Friday January 27, 2023 @03:13AM (#34543820)
Their data correlation is awful compared to competitors like Honeycomb, Dynatrace and Instana. What we want to see is something that cuts through all the noisy data and show the users what anomalies are occurring. We shouldn't be sifting through bunch of outdated dashboards and notebooks in this day and age