News for hackers. Stuff that matters.
Re:Show HN: HyperDX – open-source dev-friendly Datadog alter... by mikeshi42 on Monday September 18, 2023 @03:08PM (#37560551)
Thank you, really appreciate the feedback and encouragement!

> It seems like clickhouse is obviously a big piece of the tech here, which is an obvious choice, but from my experience with high data rate ingest, especially logs, you can run into issues at larger scale. Is that something you expect to give options around in open source?

Scaling any system can be challenging - our experience so far is that Clickhouse is a fraction of the overhead of systems like Elasticsearch has previously demanded luckily. That being said, I think there's always going to be a combination of learnings we'd love to open source for operators that are self-hosting/managing Clickhouse, and tooling we use internally that is purpose-built for our specific setup and workloads.

> I saw what is in OSS vs cloud and I think it is a reasonable way to segment, especially multi-tenancy, but do you see the split always being more management/security features?

Our current release - we've open sourced the vast majority of our feature set, including I think some novel features like event patterns that typically are SaaS-only and that'll definitely be the way we want to continue to operate. Given the nature of observability - we feel comfortable continuing to keep pushing a fully-featured OSS version while having a monetizable SaaS that focuses on the fact that it's completely managed, rather than needing to gate heavily based on features.

> on OSS, I was surprised to see MIT license

We want to make observability accessible and we think AGPL will accomplish the opposite of that. While we need to make money at the end of the day - we believe that a well-positioned enterprise + cloud offering is better suited to pull in those that are willing to pay, rather than forcing it via a license. I also love the MIT license and use it whenever I can :)

> On that note, I am curious what your target persona and GTM plan is looking like?

I think for small teams, imo the options available are largely untantilizing, it ranges from narrow tools like Cloudwatch to enterprise-oriented tools like New Relic or Datadog. We're working hard to make it easier for those kinds of teams to adopt good monitoring and observability from day 1, without the traditional requirement of needing an observability expert or dedicated SRE to get it set up. (Admittedly, we still have a ways to improve today!) On the enterprise side, switching costs are definitely high, but most enterprises are highly decentralized in decision making, where I routinely hear F500s having a handful of observability tools in production at a given time! I'll say it's not as locked-in as it seems :)

Re:Show HN: HyperDX – open-source dev-friendly Datadog alter... by addisonj on Monday September 18, 2023 @04:13PM (#37561383)
Thanks for the answers Mike!

One more follow-up on the scale side (which I mentioned with sibling comment), it isn't so much about clickhouse itself, but about scaling up ingest. From my own experience and from talking with quite a few APM players (I previously worked in streaming space), a Kafka / durable log storage kind of becomes a requirement, so I was curious if you think at some point you need a log to further scale ingest.

For enterprise side, I was previously in data streaming space and had quite a few conversations with APM players and companies building their own observability platforms, happy to chat and share more if that would be useful!

Re:Show HN: HyperDX – open-source dev-friendly Datadog alter... by mikeshi42 on Monday September 18, 2023 @04:21PM (#37561490)
Ah got it, yeah a queue of some sort is definitely useful when scaling up to buffer pre-inserted data. This is something on the OSS side we've kept open to implementation. However it's something that is highly coupled with infra footprint and internal SLA guarantees the user wants to preserve. It can range anywhere from just rely on client-side retries to setting up a HA Kafka cluster early in the ingestion pipeline.

Similar to Elastic - I think a lot of architectures are available to choose on that side when users want to scale.

Will reach out to connect!

Re:Show HN: HyperDX – open-source dev-friendly Datadog alter... by debarshri on Monday September 18, 2023 @09:41PM (#37564704)
One piece of advice here is, if you pitch yourself as a datadog competitor, then I would recommend replicating some of the GTM motions that datadog employed. For instance, you have an opportunity to go very upmarket, super enterprise orgs. You can do PLG, but ultimately every tool becomes SLG. I would recommend fine tuning that motion as that would be the one bringing larger contract 6 digit contracts and huge growth here.

I have seen orgs remove datadog because of unpredictable pricing. If you do flat price self hosted platform, you will get attention. I dont think orgs would mind hosting clickhouse. You can also bundle it with your helm charts or initial proof of concept might have lower barrier. I know some orgs have million dollar annual contracts with datadog, a cheaper more predictable priced alternative will definitely get attention.

Re:Show HN: HyperDX – open-source dev-friendly Datadog alter... by mikeshi42 on Monday September 18, 2023 @11:07PM (#37565189)
Thank you - I think that's definitely an interesting idea for us to go down for sure! We've heard a ton that the unpredictable (and insane) costs of Datadog is forcing teams to move off in droves. Something that strikes the balance between more expensive hosted solution vs cheaper but self hosted might definitely a interesting angle to try.
Re:Show HN: HyperDX – open-source dev-friendly Datadog alter... by mx20 on Tuesday September 19, 2023 @02:52AM (#37566457)
MIT License allows Amazon and other Cloud providers to offer Cloud Solutions as well. That's why most SaaS changed to AGPL or better versions that explicitly disallow Cloud offerings.
Re:Show HN: HyperDX – open-source dev-friendly Datadog alter... by datadeft on Tuesday September 19, 2023 @02:54AM (#37566474)
> While we need to make money at the end of the day

Honest question: What makes you think that you are not turning into a Datadog (price wise) once reach a certain scale?

The problem what I see with software companies that the pricing is dominated by investor requirements and when a company reaches a certain milestone change up the licensing model and the pricing with it.

Re:Show HN: HyperDX – open-source dev-friendly Datadog alter... by mikeshi42 on Tuesday September 19, 2023 @03:51AM (#37566790)
It's a classic innovator's dilemma - if/when we get there - it'd be a bit naïve of us to assume the next HyperDX isn't around the corner :) Anyone that believed in us on the way up - certainly has to believe that the same mistake will bring us down.

I'd also add that I don't think all services trend their price upwards. AWS has historically lowered prices on services and continue to offer new service-tiers with lower prices (S3 tiering as an example). As the tech matures and costs fall for our service as well, it'd be surprising if we don't do the same.

Re:Show HN: HyperDX – open-source dev-friendly Datadog alter... by datadeft on Tuesday September 19, 2023 @05:45PM (#37577248)
Fair points. Putting you forward with my current company as a potential replacement for New Relic.
Re:Show HN: HyperDX – open-source dev-friendly Datadog alter... by tmd83 on Tuesday September 19, 2023 @03:58AM (#37566823)
https://www.hyperdx.io/docs/oss-vs-cloud

This page shows event pattern available for both oss vs. cloud. The blog doesn't mention exactly how this is being which would be an interesting read but I understand if a secret sauce.

I recall quite a few years ago a standalone commercial & hosted tool for doing something like this just on logs for anomaly detection. Anyone has any reference for similar tools for working with direct log data (say from log files) or in a similar capacity like hypderdx (oss or commercial)

Re:Show HN: HyperDX – open-source dev-friendly Datadog alter... by mikeshi42 on Tuesday September 19, 2023 @05:25PM (#37576973)
It's not secret! We want to be as open as possible - and it's in the OSS version if you want to try it out.

The technical details are best explained by the authors of the original paper [1]. We weren't smart enough to come up with it on our own and can't take credit for that haha

[1] http://jiemingzhu.github.io/pub/pjhe_icws2017.pdf

Re:Show HN: HyperDX – open-source dev-friendly Datadog alter... by theharshpat on Wednesday September 20, 2023 @01:29PM (#37587044)
Interesting read! Do you have more such paper recommendations?

Where do you discover these papers to read?

Re:Show HN: HyperDX – open-source dev-friendly Datadog alter... by mikeshi42 on Thursday September 21, 2023 @05:34AM (#37595310)
Sorry for the late reply! I'd probably have to dig pretty deep into my browser history, but we did some searching around, and largely looked for some benchmark papers to get an overview and from there could easily branch out to different implementations (from the citations).

Generally pretty good way to approach research papers in a new field I recall being taught, there's always some sort of "landscape overview" paper being published that can help distill down the SoTA and you can just follow the references.