guilload

Quickwit (https://quickwit.io/) | Paris, France | Onsite and remote (based in Europe) | Full-time

The company is fully remote but we also have a small office in Paris. We prefer candidates based in Europe but can make exceptions for the right profiles.

* Senior Software Engineer 80-110k€ + 0.25-1% equity based on experience.

We’re looking for a senior software engineer to contribute to Quickwit(https://github.com/quickwit-oss/quickwit), our open-source search and analytics engine. We have an ambitious roadmap for the next 18 months (performance optimization, distributed storage, support for SQL, query optimizer, revamp of our execution engine, etc.), and this is a great opportunity to shape the future of Quickwit while tackling fun and challenging problems in the field of distributed databases.

Full job description: https://quickwit.io/jobs/distributed-software-engineer

* Developer Advocate 80-110k€ + 0.25-1% equity based on experience.

We’re looking for a Developer Advocate to take the reins of content creation and community management to boost our growth. You will get the opportunity to define our advocacy strategy and execute it. Your work will be key to the success of Quickwit, and you will get the full support of all Quickwiots.

Full job description: https://quickwit.io/jobs/developer-advocate

Contact us at: hello at quickwit dot io

We indexed Common Crawl only for the purpose of this demo so this is one-time thing, we won't deal with updates.

There are more recent versions of the dataset. We used the february/march snapshot from this year and the April snapshot just came out (https://commoncrawl.org/2021/04/april-2021-crawl-archive-now...).

We store the URI of each shard making up the index and, optionally, partition key and value(s). Along with a few flags, we also store the shard size, creation and last modification time. This additional metadata is not required for the query planning phase and is only useful for managing the life cycle of the shards and debugging/troubleshooting.

What we store on S3 is a regular tantivy index and another tiny data structure that we call "turbo index", which makes queries faster on object storages. For this demo, the tantivy indexes are fairly large and we issue HTTP Range requests against them.

https://github.com/tantivy-search/tantivy