Well, it looks like Quickwit was going to add an Enterprise license as of earlier this year (PR #5529), which I had been keeping eyes on, but this announcement says they're instead going to relicense as Apache 2.0 so the "community can continue on":

> We will be focused on building a new product with Datadog, and to ensure our open-source community can continue, we will soon release a major update of both Quickwit with a relicense to Apache License 2.0 and tantivy.

So, it looks like we'll get a more liberally licensed Quickwit, but reading between the lines suggests development of it is might otherwise be winding down? It has been pretty nice and stable in my experience, so I can't really complain much. But I was really looking forward to what else it could bring.

Congrats to the team, in any case!

"So, it looks like we'll get a more liberally licensed Quickwit, but reading between the lines suggests development of it is might otherwise be winding down?"

They will stop fulltime day-to-day effort in it themselves, probably because they have been relocated to writing a similar service but closed and integrated in DD, but it seems they want to opensource the current product with a OSI compliant license in the hopes that the community picks up the tab.

I think that's a nice trade. Could have been much worse.

By the way, also note that DD is not a total stranger in the OSS space. They actually opensourced their observability pipeline tooling for general use as Vector, which is a rock solid product. - https://vector.dev/

Vector was already OSS when they acquired the company that created it, timber.

https://www.datadoghq.com/blog/datadog-acquires-timber-techn...

And yet they did dedicate some resources on it, until now. Which basically is my point :)

They bought Vector - it was always opensource

Yes, I've been using Vector since very early on, long before Datadog acquired it, and Datadog have continued ongoing maintenance and feature additions at a slow-but-steady pace, which I think is good. Like Quickwit, Vector is very stable and already quite complete. So I'm not too unhappy.

But Vector is something that complements Datadog's offering very well, so I think that makes sense for them to be good stewards of it. Quickwit is something that somewhat actively competes against them, which is a big difference. I suspect that unlike Vector, Quickwit is probably going to stop seeing any development in pretty short order, unless the devs now can consciously go out of their way to dedicate extra hours to it.

To be clear, I think that the relicense is great, and I think it's very possible that Quickwit will be picked up/forked by someone and maintenance will continue, because it's very good, and I'd really love to see someone do metrics for it as well. So, I'm not all gloomy or anything like that.

Happy that Quickwit is going to Apache 2. Sad that the team won't be working on it anymore.

I loved that I was able to setup Quickwit on AWS lambda and have a good cloud based search engine for $0.01 / month.

Interesting comment. Any thoughts on what you can use now?

The same thing? The community will maintain it

Co-founder of Quickwit here. Seeing our acquisition by Datadog on the HN front page feels like a truly full-circle moment.

HN has been interwoven with Quickwit's journey from the very beginning. Looking back, it's striking to see how our progress is literally chronicled in our HN front-page posts:

- Searching the web for under $1000/month [0]

- A Rust optimization story [1]

- Decentralized cluster membership in Rust [2]

- Filtering a vector with SIMD instructions (AVX-2 and AVX-512) [3]

- Efficient indexing with Quickwit Rust actor framework [4]

- A compressed indexable bitset [5]

- Show HN: Quickwit – OSS Alternative to Elasticsearch, Splunk, Datadog [6]

- Quickwit 0.8: Indexing and Search at Petabyte Scale [7]

- Tantivy – full-text search engine library inspired by Apache Lucene [8]

- Binance built a 100PB log service with Quickwit [9]

- Datadog acquires Quickwit [10]

Each of these front-page appearances was a milestone for us. We put our hearts into writing those engineering articles, hoping to contribute something valuable to our community.

I'm convinced HN played a key role in Quickwit's success by providing visibility, positive feedback, critical comments, and leads that contacted us directly after a front-page post. This community's authenticity and passion for technology are unparalleled. And we're incredibly grateful for this.

Thank you all :)

[0] https://news.ycombinator.com/item?id=27074481

[1] https://news.ycombinator.com/item?id=28955461

[2] https://news.ycombinator.com/item?id=31190586

[3] https://news.ycombinator.com/item?id=32674040

[4] https://news.ycombinator.com/item?id=35785421

[5] https://news.ycombinator.com/item?id=36519467

[6] https://news.ycombinator.com/item?id=38902042

[7] https://news.ycombinator.com/item?id=39756367

[8] https://news.ycombinator.com/item?id=40492834

[9] https://news.ycombinator.com/item?id=40935701

[10] https://news.ycombinator.com/item?id=42648043

I think you forgot to add the links

Anyway tantivy is great! I love pg_search https://www.paradedb.com/blog/introducing_search (which appears to be built by another company, but on top of tantivy, which is a great feature of open source)

Now, I am worried about development being stalled after this acquisition. How does further developing tantivy in the open helps Datadog's bottom line?

I love quickwit, unfortunately datadog has a history of murdering open source (e.g. vector.io halting development and never fixing gross bugs)

Yeah, a Vector dev that is now at Datadog told me that Vector is essentially deprecated.

(Disclaimer: datadog employee)

I joined Datadog after the Vector acquisition and now currently am the manager for the Community Open Source Engineering team that works on Vector open source.

It’s def. not deprecated, but it did take awhile to sort out. It’s not easy figuring out business vs giving away software for free.

Anyways, there’s quite a few issues and GitHub discussions everyday, in addition to Discord chats.

I'm using Vector for my own infrastructure and at work, at the time it seemed the best option to ship logs to various destinations. Are there any alternatives?

If you want to check out OpenTelemetry, Otel-collector does the same job - though it's tightly coupled to opentelemetry

On the contrary, It's quite active lately - https://vector.dev/releases/

pg_search dev here -- Thanks for mentioning us.

Re: Tantivy. I'm hopeful the community Paul and the Quickwit team have built on top of Tantivy will continue to flourish. I'm sure Datadog will build product(s) with Quickwit, which is built on Tantivy and will contribute to it. Many other companies like ours (ParadeDB) and other databases also integrate it. I can't speak for others, but we'll contribute whenever possible. We're currently working on supporting nested documents in Tantivy, for example, and hoping to upstream this work.

While it's reasonable to be concerned, I'd say this is a win for Quickwit, Tantivy and, of course, the well-deserving team behind them.

Congratulations! The fact you and your team managed to built Tantivy is a huge contribution to open source.

As someone who never managed to built a fond relationship with Apache Lucene based products (Solf, Elastic). I was extremely happy to see Tantivy in open source.

BM25 scoring, proper asian language support, speed, memory foot prints, etc - amazing job! Thank you so much!

https://github.com/quickwit-oss/tantivy

IMHO Datadog made a smart move!

If Tantivy itself just stays permanently under Apache2 licence and find a sustainable path to co exist with the rest of open source community - it's all good guys. You are more than deserve a commercial success.

Congrats!!

Hoping this leads to datadog launching a logging solution that does not cost an arm and a leg at scale.

Based on my experience with them, this isn't a technical or product issue, but a cultural one. (DD is very sales-driven, in a bad way)

Honestly this is my only criticism of datadog. The product is great but good lord that pricing.

> Mezmo recently put in production Quickwit to serve thousands of customers and petabytes of logs, drastically reducing infrastructure cost and complexity while delivering the same user experience.

I can't imagine they feel great about Quickwit getting bought by a competitor after that.

It's a risk you deal with giving back to open source.

The good part for the rest of us is it's a signal that there's likely some appetite for a fork if Datadog screws the pooch.

Either they did it on their own, in which case that's open source for you, or they have some kind of contract with quickwit, in which case I'd be really surprised if they don't have some clauses which handle this.

We just moved from Elastic to Quickwit. Sigh. What other open source, object storage backed logging databases do people recommend?

Loki by Grafana Labs is nice (https://grafana.com/oss/loki/). There was a time (3+ years ago) where the product was changing pretty rapidly and much of the documentation was on git, so we had a few headaches doing minor version bumps, but I believe its much more mature now.

Just don’t look at the list of open issues and the lack of responses.

Loki is significantly different in several ways, including the lack of full-text indexing.

Loki has the following problems:

- It is non-trivial to setup and operate. It requires properly configuring and orchestrating numerous components - distributor, ingestor, querier, query frontend, compactor, consul, ruler, memcache, index service, etc. - see https://grafana.com/docs/loki/latest/get-started/architectur...

- Its' configs tend to break with every new release, since some configs become obsolete and some configs change names and/or structure.

- It is very slow on "needle in the haystack" queries such as "search for logs containing some rarely seen trace_id", since it needs to read, unpack and scan all the logs at object storage. This may be also very expensive if the object storage service charges for read IO.

- It doesn't support log fields with big number of unique values (aka high-cardinality log fields). It you'll try storing logs with high-cardinality fields into Loki, then it will quickly explode with enormous RAM usage. Recent Loki releases provide half-baked solution for this problem - "structured metadata" ( https://grafana.com/docs/loki/latest/get-started/labels/stru... ), but it is still experimental and requires non-trivial configs (aka it doesn't work).

I suggest giving SigNoz a try. Built on top of Clickhouse, it’s fast and scalable

https://github.com/SigNoz/signoz

PS: I am maintainer at SigNoz

what another open source you can recommend ?

I'm not the person you asked -- and I also want to be transparent that I only PoC-ed it and due to external circumstances didn't get it all the way out to production -- but I really like how https://github.com/metrico/qryn (AGPLv3) thinks about the world. It is, like SigNoz, unified (logs, metrics, traces) but it actually implements several of the common endpoint schemes allowing it to pretend to be "your favorite tool" which plausibly helps any integration story <https://github.com/metrico/qryn#%EF%B8%8F-query> and <https://github.com/metrico/qryn#-vendors-compatibility>. Had I gotten it up and running, I was going to contribute a Splunk ingest adapter since that was what we were trying to replace

I was going to take advantage of Clickhouse using S3 as warm-to-cold storage since my mental model is that most logs, metrics, and traces are written and not read https://clickhouse.com/docs/en/integrations/s3#configuring-s...

I believe one could do that with SigNoz, too, so I don't mean to imply that trickery was qryn specific, just that I didn't want to get into the "constantly resizing io3 PVC" game

qryn developer here, thanks for the mention and for the kind words! hope we'll get another shot in the future!

Maybe Loki, but it has some limitations related to high cardinality data and indices (https://signoz.io/blog/logs-performance-benchmark/?utm_sourc...)

Take a look at VictoriaLogs - https://docs.victoriametrics.com/victorialogs/ . It is a single small executable, which runs out of the box with the default configs (e.g. it is zero-config) and stores all the data into a single directory (support for object storage coming soon).

I hate Datadog. We use their name as an epithet at our company for how not to sell/market. Their selling tactics circa 2015-2018 completely burned us out. Endless calls and emails. The icing on the cake was an AWS reInvent presentation on Lambda right when lambda was first announced. We were pumped to get in on lambda early. Got the whole crew to attend the talk. Turned out to be a rudimentary copy of a Barr "lambda up and running" blog wrapped in a stand up comedy routine hawked by a Datadog employee who made sure to tell us he was a Datadog employee. Get us all drunk and happy and think Datadog is cool.

Genuine question: has the company changed enough in the interim to deserve a second look?

What does any of that have to do with the actual product?

The product itself is very good, but the sales process is truly awful. Random calls with non-technical reps unable to answer basic questions like, "now that you've added this to my GCP account for 2 weeks, how much is this going to cost?" They'd say they're not sure but they have a startup deal with $xxxx minimum commit for 12 months gets you two months of extra trial, cancel anytime no questions asked. It's not just bad, it's comically bad.

Different person with similar stance: Those specific examples? Whatever.

We did get absolutely burnt by other manifestations of the DataDog approach: The billing model was (is?) very much not good, transparent or predictable and staying on top of costs was close to a nightmare. The way surprise costs and contract changes (triggered by them) was handled did not feel honest.

The product itself is great but from my perspective it's absolutely not worth having to deal with their business side of things and the risks, costs (money, time, attention) and stress associated.

If I were a Quickwit customer I'd start looking for alternatives.

qw is open source so you can continue to use the foss-licensed software as it is

It made me extremely distrustful of any and all interactions I would have with an employee. Is every email I send to my rep going to turn into an upsell? Are they being straight with me in answering my question?

That is their product. You want to cancel? Send the army of clowns.

For me, as much as it pains me to admit this, the sales and account relationship process is just as important of a factor now. I'm at a level where I'm not the end user of most of the infrastructure I purchase for the business, but I'm the one that has to deal with most of the vendor interaction.

Datadog is a pain in the ass. I've got two emails and a voicemail from them just this week. We are not an active customer.

Heroku/Salesforce is also a pain in the ass. It causes enough friction with legal that I'll spend whatever effort it takes to replatform our workload just to not have to have those unending inbound calls.

NS1 was easy-peasy, but post-IBM I now receive a PDF invoice for $50 once per month with no credit card-based billing options and have to remind finance to cut a paper check. I'll be rehoming our DNS as soon as we decide on where to move it to.

tl;dr: the business experience is part of the product

There are multiple possible outcomes from the merge with Datadog.

As my ex manager once told - there is no such thing as nice people in P&L statement. Someone has to pay

It's very easy to be anxious and see the path to the dark side here.

However one of possible outcomes - there will be a valid open source competitor to Grafana ecosystems, and this along secure the rest of scene from relicensing. There is a chance it will be all win-win with clear sustainable path and no money and power struggles for founders.

I want to stay on the positive here. Time will show.

Is Grafana and its ecosystem no longer open source? I haven't used it in a while

No, no. It's all good and amazing. IMHO Grafana is getting even more important in the light of possible disappearing a solid potential competitor. Hope Datadog will be smart enough with Quickwit and opportunity to win the hearts and minds.

Clearly an opportunity for Datadog to make a big statement here, and change its perceptions.

> there will be a valid open source competitor to Grafana ecosystems

You may want to check out SigNoz [1] - takes a all in one app approach compared to different modules for each signal approach which Grafana takes

Disclaimer : I am one of the maintainers

[1] https://github.com/SigNoz/signoz

The name is perfect, Data DOG.

I have had the opposite experience. Sales has been really respectful of our time.

> Get us all drunk and happy

Damn I never get sales pitches like that.

Agreed. Once they had a whiff of our domain every engineer was being bombarded with emails like “Re: meeting next week”.

Do any engineers actually like DD? Execs/managers seem to love DD and get upsold all the time on it and ask engineers to implement some of their half-baked features. It seems like its good for alerts and dashboards for infrastructure teams. But as an engineer its a pain and using it for log analysis is much more annoying than just tracking down the actual relevant logs and grepping.

Easily the best product in its category. The value is in APM/tracing - if you're just using it for logs you're missing a lot.

If you're used to traditional Enterprise pricing it's fairly priced for the value you get but anybody coming from self-funded or VC it's very expensive. If you're already using Splunk you can afford it.

One of its best features is it's consumption priced not by seats so easy to open up for all of eng including product/QA teams and not just devs to use which is great for breaking down barriers.

It's a bit like aws in that it's a platform - use of one product tends to encourage using more from their suite.

Second this.

I used to log large apps using kibana and elastic search. Also using the clusterfuck that are all AWS tries at this (cloud watch, log insights.amd whatnot).

Nothing compares to what DD give you in observability.

Having said that, DD should only be an AWS feature. They should buy them for a couple of billions and integrate it as a service for all of AWS infrastructure.

> They should buy them for a couple of billions

DD has a market cap of $48B

Which just meant that when the business had sticker shock they just disabled all the useful features of DD which consumed so much.

Large json logs just won’t appear for hours, which makes it so fun to debug issues in deployed environments!

Showing ALL the logs isn’t cool. Showing SOME logs at random is cool.

My biggest issue with them is the absolutely, hilariously, ridiculously expensive custom metrics.

It's just a solution to garbage distributed cloud service logs and devs that don't know how to log an application.

DD is the Oracle/SAP of o11y. Scumbags that kill innovation.

It’s good, I like it. I don’t use the logs product (we splunk) but I’m big fan of the automatic profiling and trace/span stuff. It’s like 50% better UX compared to other tools I’ve tried. But it’s expensive enough we’re always thinking about moving off it. That will be a very sad day for me.

The clock begins counting down to Datadog figuring out how to charge per node, cpu, container, RAM (KB), and unicode characters indexed.

Isn't this what they already do? Log pricing is per ingested GB and monitoring is charged per node.

They have to though, right? If you look at the nonsense people generate in logs, it would be a war crime to not monetize that. 20% of the PB's of data is probably the same 2,100 UTF-16 characters of Windows event id 4624.

They already do. For example, for containers you have 5 per host included in the Pro plan (I have more in my hobby projects...) and then you pay extra for each. Their pricing model is quite complex and with time you start to wonder if all these nice features are really worth that high prices.

> This summer, the wind started to turn. We witnessed stronger open-source traction, our revenue increased dramatically, and VCs became more insistent. It was time for us to open a new chapter for the company and raise a series A round.

Rhetorically, why was it time for this?

Practically, the answer is right there: the VCs wouldn't accept a mere rapidly-growing company with great tech. It's either an up round so they can mark up the value on their portfolio, or if the market isn't hot enough for a high-priced Series A, force an exit.

Practically, what happens if the founders don’t want to play ball? VCs can muscle up and force a sale on worse terms? Just not clear how much of what goes on is asking nicely vs coercion, and where the actual leverage lies.

It’s going to depend on what the terms you agreed to are. In my experience, the company leadership are free to suggest an alternative to the hyper growth plan, but the board (which you likely won’t have a majority on) will reject it. VC’s don’t care about recouping money, they’d rather you fired everyone, pivoted, and started again and tried again with the money you have. They want successful exits, not break evens. It’s not about money it’s about reputation.

If you suggest something they don’t like, they’ll stall until you run out of money, or force you out.

You're a founder who raised seed and now appear to be hitting PMF and making repeatable sales, won't you raise A to scale?

Is that a rhetorical question, or aimed at me specifically?

Regardless, in my case, it's not my first rodeo; I didn't give up any board seats or majority control. I mostly raised from angels that I know personally and trust me. And we're a high-margin B2B business (apparently similar to Quickwit) so I don't feel particularly capital constrained without raising additional rounds.

So I don't have a board pushing me for "valuation events." But I am very familiar with this toxic dynamic from my previous experience.

> why was it time for this? Practically, the answer is right there: the VCs wouldn't accept a mere rapidly-growing company with great tech.

Remember that this is a decision you make when you accept the seed money, not when it comes to looking for further money. If you want to build a profitable growing company, bc funding is not the path to that.

Depends, if you have a majority on the board you can do whatever you want. Some companies get a seed round to then use that to bootstrap to profitability and growth over time.

Our seed round was 100% made of SAFE, so VCs did not have the power to force us to do anything.

The sentence in the blog post is a tad misleading. I suspect François is not really talking about VCs that had already invested in quickwit, but about the usual flow of other VCs who contacted us, to know about the company and be part of our eventual series A.

It just generally felt like we were "at a crossing".

No one twisted our arm.

Thanks for the clarification, and sorry for jumping to an incorrect conclusion based on vague wording. (I would edit my comment accordingly but I can't anymore.)

I integrated Quickwit into our o11y platform. It was great tech with lots of promise, and now it's dead. Yet another reason to never do business with Datahog.

How much did they get acquired for?

Sqreen was a great product for startups. Unfortunately acquired by the Datadog. Product was never fully integrated into their offerings and there is no such alternative.

Hey I'm an engineer who worked at Sqreen and now Datadog. What kind of feature is still missing in Datadog ASM? To my knowledge everything is there except managing CSP and security headers now.

Few months ago when I’ve tried it there was no full support for ruby language.

Ah yes, we are still missing RASP support on ruby, but this should come out in the next 3-6 months. Everything else should be supported though (In App WAF, playbook, user monitoring etc)

Come on this is HN, the news / link aggregator of a VC: tell us how much the exit was in $$$ ;-)