News for hackers. Stuff that matters.
Re:DataDog asked OpenTelemetry contributor to kill pull request by raybb on Thursday January 26, 2023 @06:38PM (#34540431)
Re:DataDog asked OpenTelemetry contributor to kill pull request by icepat on Thursday January 26, 2023 @06:53PM (#34540574)
Not sure I entirely follow what's going on here. Is there some context behind this that's useful to know?
Re:DataDog asked OpenTelemetry contributor to kill pull request by tonyhb on Thursday January 26, 2023 @06:56PM (#34540594)
If I understand correctly, this is a receiver — which means you can take your existing DD instrumentation in their format and have it translated to OTel using this code.

It lowers the switching cost to get off of DD.

Re:DataDog asked OpenTelemetry contributor to kill pull request by trynewideas on Thursday January 26, 2023 @07:03PM (#34540653)
And Datadog competitor Grafana sponsored it: https://twitter.com/boostchicken/status/1618692475845238784
Re:DataDog asked OpenTelemetry contributor to kill pull request by xen2xen1 on Thursday January 26, 2023 @07:09PM (#34540703)
Protect the moat at all costs!
Re:DataDog asked OpenTelemetry contributor to kill pull request by benatkin on Thursday January 26, 2023 @07:20PM (#34540798)
It's worse than that, they want security through obscurity too. They feel like someone is inappropriately tinkering with the agent they want customers to install. It's open source: https://github.com/DataDog/datadog-agent ...but the downloads are behind a login: https://github.com/DataDog/datadog-agent#datadog-agent
Re:DataDog asked OpenTelemetry contributor to kill pull request by judge2020 on Thursday January 26, 2023 @07:23PM (#34540841)
Not really - the script it gives you in-app is

  DD_API_KEY= DD_SITE="datadoghq.com" bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script_agent7.sh)"
They tell you to sign in because installing without a key leads to non-working agents and support tickets.

As for the libraries themselves, they're all on the regular package manager for that language, eg. pypi.

Re:DataDog asked OpenTelemetry contributor to kill pull request by benatkin on Thursday January 26, 2023 @07:27PM (#34540876)
Ah, that's good to know, thanks. I didn't have that link, but I can see it doesn't have an API key, so they can't change the download based on who's downloading it. That's good for transparency. Ditto the package managers. I see some Linux package manager public keys in there too.
Re:DataDog asked OpenTelemetry contributor to kill pull request by benatkin on Thursday January 26, 2023 @07:16PM (#34540767)
They want to keep their agent secret.

Secret Agent Man...

Re:DataDog asked OpenTelemetry contributor to kill pull request by angstrom on Thursday January 26, 2023 @07:26PM (#34540868)
They've given you a pull number and taken 'way your name
Re:DataDog asked OpenTelemetry contributor to kill pull request by ren_engineer on Thursday January 26, 2023 @07:24PM (#34540848)
OpenTelemetry basically allows for vendor neutral instrumentation for your metrics, logs, and traces. It makes it easy to switch to a cheaper or better service with minimal lock-in, which is obviously bad for DataDog and other companies business model
Re:DataDog asked OpenTelemetry contributor to kill pull request by MuffinFlavored on Thursday January 26, 2023 @07:28PM (#34540887)
> It makes it easy to switch to a cheaper or better service with minimal lock-in

Got any examples?

I tried running my own "stack" for a project I wanted alerting on. I landed on Jaeger all-in-one (wasted time on Zipkin, the UI just was nowhere near as good as it ought to be) Docker container in docker-compose with COLLECTOR_OTLP_ENABLED.

Re:DataDog asked OpenTelemetry contributor to kill pull request by rad_gruchalski on Thursday January 26, 2023 @09:00PM (#34541561)
I used the collector to proxy otel traces to jaeger zipkin for uniform istio tracing.
Re:DataDog asked OpenTelemetry contributor to kill pull request by xcasperx on Thursday January 26, 2023 @10:43PM (#34542188)
Check out telemetryhub.com (admittedly I work here).

We offer a free trial and don't charge per a seat.

Re:DataDog asked OpenTelemetry contributor to kill pull request by vertis on Saturday January 28, 2023 @12:00PM (#34559206)
I understand the need to charge money for things, but the $150 per month pretty much rules out any bootstrapped startups.
Re:DataDog asked OpenTelemetry contributor to kill pull request by MuffinFlavored on Sunday January 29, 2023 @01:24AM (#34565495)
I wish I knew the right answer between "spin up a cloud VPS Linux server and run your own with Docker compose" and pay $150/mo for something like DataDog
Re:DataDog asked OpenTelemetry contributor to kill pull request by m-o11y on Friday January 27, 2023 @02:39AM (#34543624)
If you're looking for something a bit more "few-clicks-and-you-are-and-running", check out OpsVerse ObserveNow: https://opsverse.io/observenow-observability/ .. Entirely powered by OSS tools, ingestion-driven pricing, and without the hassle of managing the stack and scaling up.
Re:DataDog asked OpenTelemetry contributor to kill pull request by hughesjj on Wednesday February 01, 2023 @01:12PM (#34614858)
Add another exporter to the `otel-config.yaml` and if they support it properly that should literally be all it takes. That said, "if they support it properly" is exactly what otel is working on. It's super stable for a <v1 release but is still missing some features and polish imho that prevents it from getting the official v1 tag.
Re:DataDog asked OpenTelemetry contributor to kill pull request by ChrisCooneyUK on Friday January 27, 2023 @05:08AM (#34544440)
Full transparency: I work at Coralogix HOWEVER I am speaking as me, Chris, the engineer, and do not reflect the views of Coralogix!!

This business model is only bad for a company if they're not confident that the value they bring isn't enough OR if there is a plan to ratchet up prices, once customers are locked in. If the agreement is transparent, then there's no need for this kind behaviour. That's why at Coralogix, we openly encourage our customers to use an open source integration, and we keep them around with 30 second ticket response times and 40-70% cost savings, without any performance or feature sacrifice :) https://coralogix.com/docs/opentelemetry-using-kubernetes/

Re:DataDog asked OpenTelemetry contributor to kill pull request by mindcrime on Friday January 27, 2023 @06:59AM (#34545077)
which is obviously bad for DataDog and other companies business model

Meh. The best way to keep somebody on your product is to make it easy for them to get off your product.

Re:DataDog asked OpenTelemetry contributor to kill pull request by xen2xen1 on Thursday January 26, 2023 @06:58PM (#34540617)
Wondered the same. Assume this is a threat to their business? An open source competitor can do something cool so they want to kill the functionality?
Re:DataDog asked OpenTelemetry contributor to kill pull request by hosh on Thursday January 26, 2023 @07:21PM (#34540812)
We use DD, and we have to use Otel for our Elixir apps to ingest spans into DD because there are no DD SDKs for Elixir or Erlang.

From the perspective of a customer, I can tell you that DD already has quite a bit of a moat. Their main competitive advantage, and what got us into using it, is being able to correlate data across APM, custom metrics, and logging through the use of tagging. They then densely link data together across the platform. There is also a built-in Jupyter-style notebooks. By correlating data like that, you get more value out of ingesting as much data into DD as you can. There are some additional services we're not using, such as auto-correlation with ML (and alerting for anomaly detection), and security monitoring that also looks across the entire platform using their ML tech.

Like AWS/GCP/Azure, it can get expensive, quite fast, using on-demand pricing, so there are negotiated annual contracts. Right now, our team is small, and to replicate the functionality we do use, using self-hosted open-source tooling, we might as well hire another engineer for just setting up and maintaining such a platform.

I get it that, you want to defend the moat and that eroding the little things can lead to eroding the big things. As I see it though, if you need those correlations, you'll need a certain scale and team size before it makes sense to build out something like that for yourself.

Re:DataDog asked OpenTelemetry contributor to kill pull request by aeyes on Thursday January 26, 2023 @07:37PM (#34540967)
> negotiated annual contract

Maybe we are too small but Datadog is one of the few vendors which we haven't been able to negotiate down in years. The price has always been whats on the website. I honestly don't even mind, with some vendors it feels like you are on a basar and they always tell you that their final discountns had to get approval by the CEO.

Re:DataDog asked OpenTelemetry contributor to kill pull request by hosh on Thursday January 26, 2023 @07:42PM (#34541013)
I don't know if we got a discount for an annual contract, or if the annual contract pricing was published on the website. But we were able to work out things like, using seasonal pricing because our traffic is seasonal.
Re:DataDog asked OpenTelemetry contributor to kill pull request by slyall on Thursday January 26, 2023 @08:15PM (#34541258)
I always tell people looking at Datadog to remember "DDDD" or "DataDog Don't Discount".

Compared to most Enterprise vendors it is a lot harder to get a discount from Datadog. Most vendors will give you 1/3 off just for signing a contract and committing to a spend, Datadog is not like that.

Re:DataDog asked OpenTelemetry contributor to kill pull request by shyn3 on Thursday January 26, 2023 @10:17PM (#34542043)
Just show them New Relics new pricing
Re:DataDog asked OpenTelemetry contributor to kill pull request by wink on Friday January 27, 2023 @03:04AM (#34543761)
the New Relic who either had a breach they never notified me or sold my email address? I've getting random spam to newrelic@<domain> for years now. Nope, they'll never see any money if I can avoid it.
Re:DataDog asked OpenTelemetry contributor to kill pull request by Hawxy on Friday January 27, 2023 @03:11AM (#34543816)
> Maybe we are too small but Datadog is one of the few vendors which we haven't been able to negotiate down in years.

We spend a few thousand a month with Datadog and our account manager reaches out every quarter to adjust our monthly commit up/down which provides a 20% discount (I think) or so off from the website prices.

Re:DataDog asked OpenTelemetry contributor to kill pull request by ChrisCooneyUK on Friday January 27, 2023 @05:05AM (#34544418)
Full transparency, I work for Coralogix, but customers moving from DataDog to Coralogix regularly see a 40-70% cost reduction. And we actively encourage our customers to build using OpenTelemetry because we have confidence in what the product brings :)

https://coralogix.com/docs/opentelemetry-using-kubernetes/

Re:DataDog asked OpenTelemetry contributor to kill pull request by pranay01 on Thursday January 26, 2023 @07:43PM (#34541025)
you may want to check out https://github.com/SigNoz/signoz

I am one of the maintainers. We are building a DataDog alternative with native support for opentelemetry.

Re:DataDog asked OpenTelemetry contributor to kill pull request by hosh on Thursday January 26, 2023 @07:56PM (#34541130)
I have seen it around, but didn't know it was open source. If you guys are working on correlation, then I will definitely be watching this. Consistent link urls that can be shared is also useful.

Hopefully someone else will contribute the notebooks feature. Those are very useful.

Something that DD is not careful about, is being able to consistently use UTC for all time labels in all graphs (and maybe a quick way to convert to a local time if we need to communicate with stakeholders).

(I don't know why your comment was downvoted).

Re:DataDog asked OpenTelemetry contributor to kill pull request by pranay01 on Thursday January 26, 2023 @08:11PM (#34541223)
Yeah, we are actively working on correlation. We have metrics, traces and logs in a single app - so we should be able to provide a seamless correlation.

Thanks for the point about Notebooks, we have not thought in detail on how people use that. Is it primarily to collaborate between team members when an incident happens or even when there is no incident, and you are analysing stuff

Re:DataDog asked OpenTelemetry contributor to kill pull request by hosh on Friday January 27, 2023 @11:29AM (#34547874)
We have used those notebooks for:

- Incidents, collecting different metrics and showing them next to each other, with comments

- Longer-term reliability debugging. They can form a kind of ad-hoc dashboard. These are usually issues that degrade performance, don't have immediate or wide-spread customer impact, and are things we are not immediately able to detect

- Related, performance tuning. Sometimes, the key metric is unknown. We want to explore it, and then make changes to infra, and then see if that moved the needle

- Sometimes, the ad-hoc widgets are useful enough to export to a dashboard

- I can take any widget anywhere else and import it into a notebook, or start a new notebook out of it.

The notebooks are similar to the dashboard, just that, the layout engine only allows a linear notebook layout instead of a grid. There are already text widgets, though the button to add that is easier to access. Other than the comments, it's basically a dashboard with the UI changed so that it feels like a notebook.

Keep in mind too, all dashboard and notebooks modify timestamps and other states in the browser URL, so it is easy for me to copy-paste those into Slack so that other people can see what I am seeing.

Re:DataDog asked OpenTelemetry contributor to kill pull request by SOLAR_FIELDS on Thursday January 26, 2023 @10:04PM (#34541968)
We use Datadog at the medium sized unicorn I work at. People often don’t understand how important correlating timestamps is. I wish Datadog was just opinionated as hell and said everything you see is UTC by default. Or even less dramatic than that, literally just slap the time zone on whatever it is that you’re viewing. It’s so insanely critical when you’re viewing logs on a prod outage that people immediately see and understand that the logs they are looking at are PST, CST, UTC or whatever. It’s insane that software designed to help people unfuck production systems doesn’t display that by default.

As-is we go through a song and dance whenever we look at logs and metrics “oh, this happened at X time which is Y time for most people.

Re:DataDog asked OpenTelemetry contributor to kill pull request by hosh on Friday January 27, 2023 @11:31AM (#34547909)
Our ops team standardized on UTC (though not all of our tools support that).

When we talk to stakeholders and customer-facing folks though, tend to convert it to local time.

Re:DataDog asked OpenTelemetry contributor to kill pull request by harpratap on Friday January 27, 2023 @03:13AM (#34543820)
Their data correlation is awful compared to competitors like Honeycomb, Dynatrace and Instana. What we want to see is something that cuts through all the noisy data and show the users what anomalies are occurring. We shouldn't be sifting through bunch of outdated dashboards and notebooks in this day and age
Re:DataDog asked OpenTelemetry contributor to kill pull request by grepfru_it on Thursday January 26, 2023 @07:49PM (#34541082)
This comment highlights my problem with singular project tracking issues. Once they get so massive like this, people suddenly "can't follow along". Though not specific to GitHub but in this case the ui hampers readability and the comments are very much in-the-moment. This then leaves me to ask the questions, (a) should we stop writing such mammoth comments in issues or (b) should we leverage generative AI to summarize a weekly or monthly timeline of issue interaction. What is the point of having historical issues if it's too complex to follow along? This was a question I regularly asked at GitHub that usually resulted in blank stares.

GitHub's own answer to this is to force engineers to use a /slash command to post a summary of the week's updates. Clunky, but it works.

Re:DataDog asked OpenTelemetry contributor to kill pull request by dylan604 on Thursday January 26, 2023 @08:58PM (#34541548)
but if you're actively working on a problem you're in it, and the comments probably make sense. coming in from outside via web search or shared link, yeah, it's probably hard to jump in without full context of the lead up to it. once the problem is fixed, do you really care that much after the fact and have moved on?
Re:DataDog asked OpenTelemetry contributor to kill pull request by grepfru_it on Thursday February 09, 2023 @07:27PM (#34733839)
How do you manage historical context ?
Re:DataDog asked OpenTelemetry contributor to kill pull request by neilv on Thursday January 26, 2023 @07:17PM (#34540773)
Thanks for the link. I didn't see it in first skim through. It was in hidden comments, and I had to go back and click a few before i found it.
Re:DataDog asked OpenTelemetry contributor to kill pull request by dboreham on Thursday January 26, 2023 @08:31PM (#34541362)
Is Mr boostchicken a DD employee? If so then this makes sense. Otherwise, it doesn't.
Re:DataDog asked OpenTelemetry contributor to kill pull request by capableweb on Thursday January 26, 2023 @08:55PM (#34541530)
Seems they are not (Meta is mentioned on their profile), but it doesn't make sense either way. Datadog should not be pressuring either their own employees or others to add Datadog relevant collectors to OpenTelemetry, especially when their public position they put out is that they "love" and support efforts like OpenTelemtry.
Re:DataDog asked OpenTelemetry contributor to kill pull request by bsagdiyev on Thursday January 26, 2023 @09:05PM (#34541589)
Oh wow... I worked with Boostchicken at Sony. He's not at DataDog afaik.
Re:DataDog asked OpenTelemetry contributor to kill pull request by boostchicken on Friday January 27, 2023 @04:51PM (#34552381)
I think if Borat was working at my former office, I would know about it.
Re:DataDog asked OpenTelemetry contributor to kill pull request by bsagdiyev on Saturday January 28, 2023 @09:04AM (#34557620)
Under Blake, so that side of the floor!