If a company is not already using Kafka, they wouldn’t want to maintain it “just” to have a self hosted APM system.
If I could make a recommendation to the developers of this system, it would be to focus on the interface with the streaming platform, before the implementation of using Kafka to support that interface.
Ideally, one should be able to plug in and out the queueing system of preference.
This will help adoption, and avoid coupling the success of your project to the implementation and success of Kafka.
You mentioned a great approach. Queueing system as a plugin. Thanks
I blogged my experiments with SigNoz's scale at https://signoz.io/blog/signoz-benchmarks/. Hoping to get better in fine-tuning configs and blogging.
Queuing technologies will come and go, IMO it's better to focus on the interface, and allow people to swap in whatever implementation they prefer and are accustomed to. It also benefits you in the long-term too, because an application that is less-coupled to a particular external dependency will be easier to test.
Some examples of queuing tech that's deployed successfully at scale: Redis Streams, RabbitMQ, Amazon's SQS. Since this is written in Go, you could even offer an in-memory, channel-oriented stream implementation, with no external dependencies.
Not one of these is universally better than Kafka: each offers a set of trade-offs, but a very similar interface from SigNoz's point of view.
For SigNoz's hosted/tenant-based solution, it might absolutely make more sense to use Kafka. But self-hosted users bring different trade-offs to the table, and might prefer to use another solution.
Strategically, can write/maintain the plugin for Kafka (very similar to how you operate right now, except it leaves the door open to more plugins existing in the future), and encourage community contributions for other tech. Or, when you're big enough, you might want to employ people to maintain those plugins too, since they're good for adoption.
I am trying to understand - what would be a good way to prioritise.
Yes, it'll make a big difference to adoption. If step one of your setup instructions are "provision a Kafka cluster", then you are going to lose 90% of people right there.
Ideally, your dev install is super simple and has a built-in in-memory queue thing. The key here, is to make it as simple as possible to get started. Once people have tried it out, and become invested in you, then you can say "for production scale, use Kafka instead of FastQ/SimpleQue/Whatever".
They key to that second step, is to have your product abstract the queue functionality it needs, into an interface that it uses to talk to the queue - allowing people to swap out queue backends with a simple configuration change.
So, make it simple to get started - and simple to scale up when you decide to.
Right now, we can run SigNoz with all components including Kafka and Druid in 4GB memory supporting around 200 events/sec. Though, will need to check whether this micro setup passes a run of a few days.
- [Apache Pulsar|https://pulsar.apache.org]?
My use of Kafka was as a "system of record", and attaching connectors to create views into the data from there.
I could replay a Kafka topic into a MongoDB, run some analysis, and destroy the MongoDB instance.
The journey to feeling good with Kafka was difficult, but I was too stubborn to let us give up. :)
We observed enterprise and other Observability SaaS vendors have some scripts and controllers to keep running these components. We plan to open-source that too. As you rightly pointed out running OSS needs man hours and we will try to remove those frictions.
Also when working with Prometheus and Jaeger, we observed people anyhow have to use Kafka to handle scale and mostly OSS are good at start but become pretty complicated at handling scale. Eg, Prometheus long term storage solution is Cortex which itself is difficult to manage. In that case, Kafka should be better beast to handle than multiple moving components inside Cortex. We built SigNoz as a scalable alternative inspired from stream processing architecture.
We will also be proving sampling strategies including tail-based sampling to retain important data and not unnecessarily clogging disks.
I can't speak to Druid but I'm always puzzled when I hear about Kafka being used for metrics. Most metrics are timestamped and also support being calculated in ways that support out of order handling.
It's true that people are being gouged on storage markup by monitoring companies but I don't think this particular approach is the solution. Obsessing over storage and querying costs isn't a good starting point for a startup so maybe driving good habits (stop collecting so much junk, keep it around less frequently, etc.) is a better route to help people save quiche. Either way good luck!