charliejuggler

We've been building some systems for clients recently including Moody's using Lucene-based engines for the R-part - the G part tends to be OpenAI or some such service but there's also appetite for internally hosted LLMs. The trick is good measurement, as I explained in this talk at State of Open Con. https://www.youtube.com/watch?v=Ghbd1RkNgpM

You could try the method we used for our vector search demo for e-commerce (all open source, natch) - use CLIP to get vector embeddings for product pictures and then use these for boosting or matching. https://opensourceconnections.com/blog/2023/03/22/building-v... Our demo works pretty well for searches like 'blue network cable' when the colour isn't always explicitly mentioned in the product data.

You might like the 'Which Search Engine?' panel I ran at Buzzwords earlier this year with some of the leading contenders (Vespa, Qdrant, Elastic, Solr, Weaviate) https://www.youtube.com/watch?v=iI40L4wMtyI - vector search was obviously part of the discussion

My colleague Nate wrote about using Jaccard for search engine regression testing: https://opensourceconnections.com/blog/2021/07/23/jaccard-in... If you make changes to a search algorithm and the order of results changes too much this can alarm users, so this is a good way to reduce risk.

People have been predicting the death of keyword search for at least as long as I've been working in the field (23 years and counting!). We've seen outrageous marketing claims, buzzwords like "concept search", "insight engines", many new companies promising a step change in search quality, some of which are still in business but many who shone briefly then vanished. The concept of an easy to use, fire-and-forget, scalable search engine that gives you great relevance out of the box isn't new. Yet the bag-of-words model remains the standard across the sector, it's well understood with many powerful and scalable open & closed source implementation options.

To make search work in practice however is hard. It's as much about process and people as it is about technology: many companies aren't even measuring search quality, recording search issues correctly or have an active search team (bigger than one poor overworked search person). No matter how clever the tech, these problems aren't going away: they're compounded by bad source data quality, misunderstandings of user search intent and bad search UX. Martin White, author of many books on search, describes search as a 'wicked problem'. Getting all these parts working in harmony so you can truly own your search is what we do here at OSC and it takes time, investment and commitment.

I think Vectara is very interesting and the people involved have impressive track records (there's also some other great engines like Vespa, Pinecone, Qdrant, Weaviate...). However I think the future of search is hybrid - we'll see keyword search still there for many use cases but enhanced by vector/neural approaches (the most widely used search engine Lucene recently gained vector features and work is happening on how to combine these with keyword ranking). No one approach will solve everyone's search problems, cope with special cases like part number search, or the specialised language used in some sectors, or always understanding the searcher's intent, magically without considering the human factors above or without extra tuning/training.

That said, with all these exciting new approaches, tools and companies, it's a very interesting time in the search world!

Further reading/viewing: at the Haystack EU search conference a couple of weeks ago www.haystackconf.com Dmitry Kan, host of the Vector Podcast (he featured the Vectara team a while ago) gave a great keynote describing the current state of vector search - I wasn't going to release the video until Monday but you can get an early look here https://youtu.be/2o8-dX__EgU . You can also read the joint article we wrote for The Search Network on vector search here https://opensourceconnections.com/wp-content/uploads/2022/05... (aimed at executives and others needing to understand the field).

If you're interested in the detail behind this check out Simon Hughes' talk at our Haystack search relevance conference last year https://haystackconf.com/2021/ - the video is here https://www.youtube.com/watch?v=bWc4DOM-CHk . Haystack US 2022 is open for business (and talk submissions) too at www.haystackconf.com if you're into search!