News for hackers. Stuff that matters.
Re:Find anything fast with Google's vector search technology by hiddencost on Tuesday December 14, 2021 @04:16PM (#29557907)
I recommend readers take parent post with a grain of salt.

(1) Google's offering returns with-in <5ms, in my experience. (2) the demo is for paragraphs, not short text. You're putting mismatched data into the input, of course it's not going to work. Try a paragraph as suggested.

Re:Find anything fast with Google's vector search technology by freediver on Tuesday December 14, 2021 @04:36PM (#29558225)
Hmm.. There is no web service that returns response in <5ms, unless you are sitting at the very terminal of the hardware producing the output.

The demo featured in this PR takes about 800-1000ms total to produce search results. How much of that is the actual API is not known. Typically, https request to an API in the cloud will cost you at least 50ms of network latency, more likely 100ms-200ms. If you are running vector search on premise you will obviously not have this overhead.

Text embeddings typically work for short text as well as paragraphs (paragraph embeddings are usually mean/max of word embeddings anyway) simply because most commercial use cases demand handling of short text input (because nobody is inputting a paragraph into a typical search box; what use is a news search if you can not type a single word like 'Biden' or 'gpt3' into it).

Re:Find anything fast with Google's vector search technology by rrdharan on Tuesday December 14, 2021 @06:08PM (#29559415)
> Hmm.. There is no web service that returns response in <5ms, unless you are sitting at the very terminal of the hardware producing the output.

Typical Google Cloud Bigtable P(50) is <= 4ms to a GCE VM in the same region.

Source: I work on Google Cloud Bigtable.

EDIT: I should clarify that applies to point reads on an established connection using gRPC-over-HTTP/2 from e.g. Java/golang/C++ client, and doesn’t include cold connection set-up / uncached auth / TLS negotiation, but those are well into P(99.999) territory if you’re doing millions of QPS (which applies to most of our large customers).

Re:Find anything fast with Google's vector search technology by rrdharan on Tuesday December 14, 2021 @06:19PM (#29559519)
Also pretty sure customers using hosted Redis / Memcached offerings like Google Cloud Memorystore or AWS Elasticache etc. get sub-1ms times for service-to-VM responses. Those aren’t over HTTP of course… so perhaps they don’t count as “web services”?
Re:Find anything fast with Google's vector search technology by freediver on Tuesday December 14, 2021 @06:33PM (#29559651)
I think you are talking about latency over the equipment in the same datacenter/intranet here which, yes, I do not count as web services but are more akin to 'sitting at the terminal'. If I could access Google Cloud Bigtable over the web from my home Mac at <5ms, that would of course be the biggest thing since the invention of the web :)
Re:Find anything fast with Google's vector search technology by hiddencost on Tuesday December 14, 2021 @04:47PM (#29558376)
It's a cloud offering, so the machines are located near each other. Using it with other cloud services is a fair comparison to running it in the same box on-prem.

The offering is similarity search, not a search engine. They offer image to image as another comparison point.

Re:Find anything fast with Google's vector search technology by freediver on Tuesday December 14, 2021 @04:51PM (#29558441)
With the caveat of having to use GCP to host your server too, I can agree with you (although 5ms still sounds incredibly low, how many vectors was that?).

I was obviously talking about a general use case where a user considers using an API like this vs running Faiss, and their server can be anywhere (a use case that is more common to me personally).

Re:Find anything fast with Google's vector search technology by pierrefermat1 on Tuesday December 14, 2021 @11:34PM (#29562130)
We are only talking about backend performance so I don't really think it's fair to include network latency on this metric, it can also hugely varied based on your current network/location setup
Re:Find anything fast with Google's vector search technology by freediver on Tuesday December 14, 2021 @11:58PM (#29562277)
> For low-latency (up to couple of hundred milliseconds) use cases, it is highly unlikely that any cloud solution like this would be a better choice than something deployed on premise because of the *network overhead*.