ankitnayan

and here query builder helps by making it earier to do cross-signal joins and subqueries. I see that is upcoming in SigNoz https://signoz.io/blog/query-builder-v5/#what-we-couldnt-shi...

You might also like the interactive dashboard feauture that was released in the recent launch week https://www.youtube.com/watch?v=YQTQXq0F5Iw&ab_channel=SigNo...

I think all of us agree that OpenTelemetry's end-goal of making Observability vendor neutral is futuristic and inevitable. We can complain about it being hard to get started, bloated, etc but the value it provides is clear, esp, when you are paying $$$ to a vendor and stuck with it.

OpenStandards also open up a lot of usecases and startups too. SigNoz, TraceTest, TraceLoop, Signadot, all are very interesting projects which OpenTelemetry enabled.

The majority of the problem seems like sentry is not able to provide it's sentry like features by adopting otel. Getting involved at the design phase could have helped shaped the project that could have considered your usecases. The maintainers have never been opposed to such contributions AFAIK.

Regarding, limiting otel just to tracing would not be sufficient today as the teams want a single platform for all observability rather than different tools for different signals.

I have seen hundreds of companies switch to opentelemetry and save costs by being able to choose the best vendor supporting their usecases.

lack of docs, learning curve, etc are just temporary things that can happen with any big project and should be fixed. Also, otel maintainers and teams have always been seeking help in improving docs, showcasing usecases, etc. If everyone cares enough for the bigger picture, the community and existing vendors should get more involved in improving things rather than just complaining.

you can add https://github.com/open-telemetry/opentelemetry-collector-co... at signoz's otel-collector which will scrape your service's endpoint periodically. If your service is down, this will give 5xx error and you can set an alert on that.

Another alternative is to use an alert to notify on a metric being absent for sometime. Both of these should work

Interesting post.

How do you apply restrictions on your queries? Otherwise a few concurrent queries scanning huge data or being slow due to groupby, etc can slowdown the system.

Also, I see a sorting key of `ORDER BY (PodName, Timestamp)`. While debugging, filtering by service_name, deployment_name, env, region, etc is probably going to be slow?

However the CPU and memory resources used are very minimalist. On no load conditions, it takes around 0.3-0.5 CPUs and 1.5-2GB RAM

I have seen it ingest 500K events/s. How did you conclude the poor perf?

When handling surges of the order of 10x, it's much more difficult to scale the different components of loki than to write them to Kafka/Redpanda first and consume at a consistent rate.

it's very hard to think s3 work as a buffer. Every datastore can work for almost all storage usecases buffer/queue/db when the scale is low but the latter were designed to work at scale

Caching is to improve read performance whereas Kafka is used to handle ingest volume. I couldn't correlate the Grafana articles shared

Hi Prabhat, I am one of the maintainers at SigNoz. Nice to see OpenObserve's belief that the future of Observability should be OpenSource. We chose clickhouse rather than building a database as it takes multi-year effort to move a db to maturity and clickhouse has been battle-tested at Yandex, Uber and Cloudflare. Clickhouse also provides native integration with s3 and other blob storages. Our users have been using disk as hot storage for a week and moving the data to s3 after that. Tiered storage is really cool in terms of query performance.

We have also seen logs data of our users at a compression ratio of 30x/40x. We have published a logs benchmark (https://github.com/SigNoz/logs-benchmark) where the data is very high-cardinal (causing a compression factor of only 2.5x). Would love to see how does OpenObserve perform in that dataset someday.

Wishing you best for the journey ahead.

An OpenSource alternative to DataDog built on ClickHouse database which is also native to OpenTelemetry https://github.com/SigNoz/signoz

Yes...we are avoiding join altogether. Currently we used join in timeseries but we are probably moving away from that due to perf. Single table is amazingly fast

Basically, the choice of DB is different, relational vs analytical DB. IMO clickhouse should be better at ingestion rate and aggregation queries.

What does your schema for metrics look like? Using materialised views? I am particularly interested in storing metrics with labels(key/val pairs) Eg, Prometheus metrics. You can't flatten them out into columns due to high dimensionality(can need millions of columns). Do you store the labels in an array?

I would love to see some benchmarks when making such a heavy claim. I would be interested in knowing performance of ingestion rate, query timings and resource usage.

We have recently been talking to a few companies in HealthCare and Fintech, and it seems they have strong needs for data to remain on-premise. One of such verticals in the company is around monitoring, logs and traces. They may contain personal information and these companies want to keep it under their own control. Hence, they can't send data to SaaS vendors due to regulations around data sovereignty.

They cannot use DataDog, NewRelic and onPrem Dynatrace & AppDynamics are closed source. A recent good alternatives are Prometheus for Metrics, Jaeger/Zipkin for distributed tracing and Loki for logs. First of all setting up all these take months and running a scaled version of these OSS is pretty complex.

We are building SigNoz (https://github.com/SigNoz/signoz) to address these above problems. SigNoz is an open source alternative to DataDog. SigNoz has Kafka and Apache Druid, both of them are industry tested highly scalable components. Druid powers analytical capabilities on tracing data which is missed in both Jaeger and Zipkin.

Below are the motivations for building the project:

- Out-of-box features that other OSS miss. Get started in minutes and not months

- Seamless integration of Metrics and Traces in a single dashboard

- SaaS vendors charge an insane amount to provide Application Monitoring. They often surprise you with huge month end bills without any transparency of data sent to them

- Data privacy and compliance demands data to not leave the network boundary

- Highly scalable architecture

- No more magic happening in agents installed in your infra. You take control of sampling, uptime, configuration

- Build modules over SigNoz to extend business specific capabilities

- Full support for OpenTelemetry, which is a vendor neutral instrumentation framework

We would love to know your thoughts on SigNoz

hmm..got your point. We shall definitely look into other queuing system to be integrated as interface. Trying to understand better, what's a super simple dev setup like (to get the adoption)?

Right now, we can run SigNoz with all components including Kafka and Druid in 4GB memory supporting around 200 events/sec. Though, will need to check whether this micro setup passes a run of a few days.

Thanks, feel free to reachout to us if you have any concerns or want to discuss in general. We are always eager to learn and help.

really liked the way you put things to clarity. Thanks for these inputs and suggestions, will definitely think harder on this.

Nothing wrong there. If enough users want, we can add clickhouse also

Correct, Ideally monitoring stack should be outside the blast radius of other applications. Will handling another Kafka cluster (probably smaller than business Kafka) be a pain for the team given the team already knows managing one business Kafka. What do you think?

Grafana, for long, has been used to monitor time-series data and recently has been moving towards observability (including traces and logs). We are different in quite a few fronts.

1. There are specific observabilty specific UI widgets like serviceMap, SLOs and error budgets, I don't know whether Grafana provides it now. Also, last I used Grafana, linking and moving from one dashboard to another is still a pain. You can get a better idea of how different observability UI can get from Grafana by looking into LightStep demo.

2. We can run aggregated on filtered traces. Eg, I can get 99th percentile response time of a tag say payment_channel. am afraid this can be extracted from traces by Grafana.

3. SigNoz is easily extendible by adding your stream processing application to slice n dice data in your own way

hmm..I get your point. I searched for Kafka alternatives for a bit before including it on our stack. Though, I couldn't find something more adopted by all. It would be good to know a few Kafka alternatives you prefer which can handle equivalent production scale?

I completely agree with you. For companies not already using Kafka, this will ask for a big commitment to self-host Kafka.

You mentioned a great approach. Queueing system as a plugin. Thanks

Great point. To start off we shall provide different hardware configs like micro, small, medium, large, xlarge with the scale that they can handle.

We soon plan to emit metrics from different components of SigNoz and setup autoscaling of different components. Druid has already put some thought in autoscaling. Checkout https://druid.apache.org/docs/latest/configuration/index.htm... and https://www.adaltas.com/en/2019/07/16/auto-scaling-druid-wit...

I was pretty much surprised to see the results too. A single node Kafka with 2GB as xmx value, was ingesting at 4500 events/sec (around 1MB/s) on a single partition.

I blogged my experiments with SigNoz's scale at https://signoz.io/blog/signoz-benchmarks/. Hoping to get better in fine-tuning configs and blogging.