News for hackers. Stuff that matters.
Re:Indexing one petabyte of logs per day with Quickwit by dracyr on Saturday March 23, 2024 @02:18AM (#39797859)
Never had the chance to use Quickwit at a $DAYJOB (yet?), but I really appreciate the fact that it scales down quite well too. Currently running it on my homelab, after a number of small annoyances using Loki in a single-node cluster, and it's been working very well with very reasonable resource usage.

I also decide to use Tantivy (the rust library powering/written by Quickwit) for my own bookmarking search tool by embedding it in Elixir, and the API and docs have been quite pleasant to work with. Hats of to the team, looking forward to what's coming next!

Re:Indexing one petabyte of logs per day with Quickwit by francoismassot on Saturday March 23, 2024 @03:08AM (#39798065)
Some companies are using it with AWS Lambda to scale to 0.
Re:Indexing one petabyte of logs per day with Quickwit by tecleandor on Saturday March 23, 2024 @08:09AM (#39799327)
Ah Loki, I wanted to try it at my homelab bit it wasn't as simple as it says. Now I wanted to try Zincsearch or Openobserve. Have you tried that?
Re:Indexing one petabyte of logs per day with Quickwit by netingle on Saturday March 23, 2024 @08:48AM (#39799560)
> it wasn't as simple as it says

mind elaborating? we built loki for some pretty massive scale but I've always tried to make it work at super small scale to. what went wrong?

Re:Indexing one petabyte of logs per day with Quickwit by pranay01 on Saturday March 23, 2024 @09:07AM (#39799653)
You might want to have a look at SigNoz [1] as well. We have also published some perf benchmark wrt Elastic & Loki [2] and have some cool features like logs pipeline for manipulating logs before ingestion

[1] https://github.com/signoz/signoz [2] https://signoz.io/blog/logs-performance-benchmark/

Re:Indexing one petabyte of logs per day with Quickwit by bbkane on Saturday March 23, 2024 @11:48AM (#39800761)
I use OpenObserve and I quite enjoy it
Re:Indexing one petabyte of logs per day with Quickwit by mdaniel on Saturday March 23, 2024 @12:44PM (#39801218)
in case it matters to others, https://github.com/openobserve/openobserve/tree/v0.7.0 is the last Apache2 licensed copy before they went AGPL with 0.7.1

https://github.com/openobserve/openobserve/blob/v0.7.0/.env.... is some "onoz" for me, but just recently someone submitted https://github.com/aenix-io/etcd-operator to the CNCF sandbox so maybe things have gotten better around keeping that PoS alive

Re:Indexing one petabyte of logs per day with Quickwit by nextaccountic on Sunday March 24, 2024 @12:46PM (#39808324)
Tantivity is great!

Here is a postgres extension that uses it to provide full text search

https://blog.paradedb.com/pages/introducing_bm25

https://news.ycombinator.com/item?id=37557127

Re:Indexing one petabyte of logs per day with Quickwit by francoismassot on Sunday March 24, 2024 @05:34PM (#39810722)
tantivy, not tantivity!!!!!