> It seems like clickhouse is obviously a big piece of the tech here, which is an obvious choice, but from my experience with high data rate ingest, especially logs, you can run into issues at larger scale. Is that something you expect to give options around in open source?
Scaling any system can be challenging - our experience so far is that Clickhouse is a fraction of the overhead of systems like Elasticsearch has previously demanded luckily. That being said, I think there's always going to be a combination of learnings we'd love to open source for operators that are self-hosting/managing Clickhouse, and tooling we use internally that is purpose-built for our specific setup and workloads.
> I saw what is in OSS vs cloud and I think it is a reasonable way to segment, especially multi-tenancy, but do you see the split always being more management/security features?
Our current release - we've open sourced the vast majority of our feature set, including I think some novel features like event patterns that typically are SaaS-only and that'll definitely be the way we want to continue to operate. Given the nature of observability - we feel comfortable continuing to keep pushing a fully-featured OSS version while having a monetizable SaaS that focuses on the fact that it's completely managed, rather than needing to gate heavily based on features.
> on OSS, I was surprised to see MIT license
We want to make observability accessible and we think AGPL will accomplish the opposite of that. While we need to make money at the end of the day - we believe that a well-positioned enterprise + cloud offering is better suited to pull in those that are willing to pay, rather than forcing it via a license. I also love the MIT license and use it whenever I can :)
> On that note, I am curious what your target persona and GTM plan is looking like?
I think for small teams, imo the options available are largely untantilizing, it ranges from narrow tools like Cloudwatch to enterprise-oriented tools like New Relic or Datadog. We're working hard to make it easier for those kinds of teams to adopt good monitoring and observability from day 1, without the traditional requirement of needing an observability expert or dedicated SRE to get it set up. (Admittedly, we still have a ways to improve today!) On the enterprise side, switching costs are definitely high, but most enterprises are highly decentralized in decision making, where I routinely hear F500s having a handful of observability tools in production at a given time! I'll say it's not as locked-in as it seems :)
One more follow-up on the scale side (which I mentioned with sibling comment), it isn't so much about clickhouse itself, but about scaling up ingest. From my own experience and from talking with quite a few APM players (I previously worked in streaming space), a Kafka / durable log storage kind of becomes a requirement, so I was curious if you think at some point you need a log to further scale ingest.
For enterprise side, I was previously in data streaming space and had quite a few conversations with APM players and companies building their own observability platforms, happy to chat and share more if that would be useful!
Similar to Elastic - I think a lot of architectures are available to choose on that side when users want to scale.
Will reach out to connect!
I have seen orgs remove datadog because of unpredictable pricing. If you do flat price self hosted platform, you will get attention. I dont think orgs would mind hosting clickhouse. You can also bundle it with your helm charts or initial proof of concept might have lower barrier. I know some orgs have million dollar annual contracts with datadog, a cheaper more predictable priced alternative will definitely get attention.
Honest question: What makes you think that you are not turning into a Datadog (price wise) once reach a certain scale?
The problem what I see with software companies that the pricing is dominated by investor requirements and when a company reaches a certain milestone change up the licensing model and the pricing with it.
I'd also add that I don't think all services trend their price upwards. AWS has historically lowered prices on services and continue to offer new service-tiers with lower prices (S3 tiering as an example). As the tech matures and costs fall for our service as well, it'd be surprising if we don't do the same.
This page shows event pattern available for both oss vs. cloud. The blog doesn't mention exactly how this is being which would be an interesting read but I understand if a secret sauce.
I recall quite a few years ago a standalone commercial & hosted tool for doing something like this just on logs for anomaly detection. Anyone has any reference for similar tools for working with direct log data (say from log files) or in a similar capacity like hypderdx (oss or commercial)
The technical details are best explained by the authors of the original paper [1]. We weren't smart enough to come up with it on our own and can't take credit for that haha
Where do you discover these papers to read?
Generally pretty good way to approach research papers in a new field I recall being taught, there's always some sort of "landscape overview" paper being published that can help distill down the SoTA and you can just follow the references.
So often that means you need something like a Kafka to get the bulk ingest to really perform to get batch sizes large enough.
That kind of gets into one of the challenges of OSS observabilility systems, you don't want to make the dependencies insane for someone who only has a few thousand logs a second, but generally at some point of scale you do need more.
We ran into some challenges with async inserts at highlight.io [2]. Namely, ClickHouse Cloud has an async flush size configured (that can't be changed AFAIK) that isn't large enough for our scale. Once you async insert more than can be flushed, you get back pressure on your application waiting to write while Clickhouse flushes the queue. We found that implementing our own batched flushing via kafka [3] is far more performant, allowing us to insert 500k+ RPS on the smallest cloud instance type.
[1] https://clickhouse.com/docs/en/optimize/asynchronous-inserts [2] https://github.com/highlight/highlight/tree/main [3] https://github.com/highlight/highlight/blob/4d28451b1935796d...
I wonder why not Apache Druid
No. Clickhouse is opensource with Apache License [0].
[0] - https://github.com/ClickHouse/ClickHouse/blob/master/LICENSE
Clickhouse runs as a single binary and is familiar sql/olap.
The background of someone with a DX background comes through! I will be looking into this a lot more.
Here are a few comments, notes, and questions:
* I like the focus on DX (especially compared to other OSS solutions) in your messaging here, and I think your hero messaging tells that story, but it isn't reinforced as much through the features/benefits section
* It seems like clickhouse is obviously a big piece of the tech here, which is an obvious choice, but from my experience with high data rate ingest, especially logs, you can run into issues at larger scale. Is that something you expect to give options around in open source? Or is the cloud backend a bit different where you can offer that scale without making open source so complex?
* I saw what is in OSS vs cloud and I think it is a reasonable way to segment, especially multi-tenancy, but do you see the split always being more management/security features? Or are you considering functional things? Especially with recent HashiCorp "fun" I think more and more it is useful to be open about what you think the split will be. Obviously that will evolve, but I think that sort of transparency is useful if you really want to grow the OSS side
* on OSS, I was surprised to see MIT license. This is full featured enough and stand alone enough that AGPL (for server components) seems like a good middle ground. This also gives some options for potentially a license for an "enterprise" edition, as I am certain there is a market for a modern APM that can run all in a customer environment
* On that note, I am curious what your target persona and GTM plan is looking like? This space is a a bit tricky IMHO, because small teams have so many options at okay price points, but the enterprise is such a difficult beast in switching costs. This looks pretty PLG focused atm, and I think for a first release it is impressive, but I am curious to know if you have more you are thinking to differentiate yourself in a pretty crowded space.
Once again, really impressive what you have here and I will be checking it out more. If you have any more questions, happy to answer in thread or my email is in profile.