vadman97

If there was only one alert criteria, that'd be simple. Our alerts can be configured for any data filters (eg. only matching logs with column `level='error'`); we would have to create a unique MV for each alerts' filter condition.

We insert from our alerts worker because we want the aggregation to happen per alert (with the aggregated data filtered by the particular alert definition). As each alert is evaluated, we run the following [1] INSERT INTO ... SELECT ... statement based on the alert definition. We can't aggregate with an MV since we'd need to create an MV per unique alert that a customer may set up.

[1]: https://github.com/highlight/highlight/blob/c526daea31fdf764...

Forget the exact details, but we ran into issues with tuning the two setting you're describing on the Cloud instance (these settings were user level settings that seemed to get reset by ClickHouse Cloud because they were updated based on the cluster size). Perhaps this could change with Cloud and isn't an issue with the OSS version.

Once we hit >100k inserts per second, async inserts didn't work well for us because we had limited control over the background async insert batching happening on the cluster. The background inserts would be too small, resulting in many merges running, causing high CPU, causing high back-pressure (latency) on async inserts which would just result in an ingestion delay.

Plus, async inserts are only available on ClickHouse Cloud.

We do via the OpenTelemetry Elixir SDK (and the Phoenix extension) https://opentelemetry.io/docs/languages/erlang/

Materialized views can help change the ORDER BY with 0 downtime:

* Create a new version of the table with the new ORDER BY.

* Create a materialized view that will insert from the old table to the new table.

* Update your application code to query the new table.

* Start inserting data into the new table.

Exploring bloom filter index merges would be an interesting addition. I do wish it were easier to profile merge performance to break down where most of the CPU time is being spent.

3 nodes of 120 GiB and 30 vCPU each.

I see what you did there.

Our browser client would help with tracing what network requests are being made by said apps since it would capture all network requests. However, Shopify may restrict what our browser client can do in your frontend store (I'm guessing that the 3rd party apps are added to a shop as iframes or are otherwise sandboxed).

Hey HN, I'm Vadim, the CTO at highlight.io. We offer session replay to engineering teams and have had a lot of questions around client-side performance.

We just published a benchmark along with a short blog post covering the results. If you'd like to try it yourself, check out https://github.com/highlight/session-replay-performance-benc...

Presumably, they mean millions of logs a day when saying "millions of events a day", of which some subset corresponds to errors.

Our errors ingest is quite affordable (https://www.highlight.io/pricing) and let's you filter out data you are not interested in ingesting. Our logs ingest is even cheaper because of the underlying ClickHouse storage we use.

Regarding model adaptation, we haven't yet explored a fine-tuned model, but it makes a lot of sense for a given class of errors. For a given code-base that is using highlight, the errors will typically be of a given language / infrastructure, so fine-tuning the model to those errors should be beneficial.

As for predicting potential errors, this will particularly make sense as an anomaly detection mechanism across metrics and logs. A class of 'important' errors based on the LLM's understand of the error, as well as historical comparison to normal error rates, is something we're exploring with an 'interesting errors' concept - stay tuned for more there!

> Have you looked into finetuning the embedding model to your use case?

Not yet, though this is definitely one of the next steps for us. The `gte-large` model we use is trained on a variety of text, but a hypothesis is one trained or fine-tuned on technical / code-related content may work better.

> Have you considered faster foundation models?

Any in particular that you would suggest? We're still pretty new to this so would love to learn about other recommendations. Would a foundation model perform as well at this task?

> How far could you go with this idea? Could this be the basis of a new monitoring platform?

Certainly; there are traditional ML approaches that could be applied to monitoring as well and we're heavily exploring this (ie. for metric anomaly detection). Another area we're exploring embeddings grouping is for filtering ingest to help folks only ingest / retain data that they actually want, but without the overhead of strict filter rules. Tons more to explore in this space, and you will certainly be hearing more from us here.

Agreed, but the argument for causing damage still requires a human actor to execute GPT-4's advice. Should we be censoring LLMs just because we are offended by the output or find it misleading? There is plenty of misinformation on the internet already; in my opinion, the power to deal damage is still in the hands of humans.

ClickHouse Async insert docs [1].

We ran into some challenges with async inserts at highlight.io [2]. Namely, ClickHouse Cloud has an async flush size configured (that can't be changed AFAIK) that isn't large enough for our scale. Once you async insert more than can be flushed, you get back pressure on your application waiting to write while Clickhouse flushes the queue. We found that implementing our own batched flushing via kafka [3] is far more performant, allowing us to insert 500k+ RPS on the smallest cloud instance type.

[1] https://clickhouse.com/docs/en/optimize/asynchronous-inserts [2] https://github.com/highlight/highlight/tree/main [3] https://github.com/highlight/highlight/blob/4d28451b1935796d...

How do you think about the query syntax? Are you defining your own or are you following an existing specification? I particularly love the trace view you have, connecting a frontend HTTP request to server side function-level tracing.