egorr

personal site, some photos and stuff i worked on: https://intest.dev/

hey, supabase engineer here; we didn’t check that out with files that small, but thanks for the idea, i will try it out

the only thing i can say related to the topic is that s3 multipart outperforms other methods for files larger than 50mb significantly, but tends to have similar or slightly slower speeds compared to s3 regular upload via supabase or simplest supabase storage upload for files with size about and less than 50mb.

s3-multipart is indeed the fastest way to upload file to supabase with speeds up to 100mb/s(115 even) for files>500mb. But for files about 5mb or less you are not going to need to change anything in your upload logic just for performance cause you won’t notice any difference probably

everything mentioned here is for upload only

interesting quote: "A vital component of Foundations is its robust and ergonomic API for seccomp, a Linux kernel feature for syscall sandboxing. This feature enables the setting up of hooks for syscalls used by an application, allowing actions like blocking or logging. Seccomp acts as a formidable line of defense, offering an additional layer of security against threats like arbitrary code execution."

hey hn, supabase engineer and blogpost coauthor here.

we made our first experiments with HNSW index for vector search and noticed about 5 times better performance compared to IVF with high dimensional vectors, such as embedding-ada by OpenAI.

we haven’t included results for smaller models like gte-small (384d) just yet, but we’re currently running those benchmarks as i write this comment. in our smoke tests the difference in performance isn’t as pronounced, but it still appears promising, suggesting that switching to HNSW could be beneficial for the majority of use cases.

oh there have been improvements with IVF as well, with index building times decreasing by roughly half. So, if you’re considering sticking with IVF, it’s advisable to upgrade to the latest version for these enhancements.

you can find extended version of ann testing framework in the GitHub [0]. its just the original one with a lil bit of code for pgvector and vecs lib.

[0] https://github.com/egor-romanov/vector-db-benchmark/tree/fea...

even networking alone adds about 1ms or so, and these are 2 hops each side: lb - sv - db, instead of the same instance for pgb scenario

another 1ms goes for a gain of a millions connections, multitenancy, and a lot of features that are possible to implement and are planned to implement.

i am curious what usecase do you have, that 2ms is unacceptable? in most cases it would be insignificant to network latency to the end client

supabase engineer here

would be happy to answer questions about load testing setup

here is the github repo[0] with an app that helps me launch terraforms and k6 and keep track of launches/results

[0]https://github.com/supabase/benchmarks

should be yes, but even in examples from openai, they usually do splitting into chunks

For example in chatgpt-retrieval-plugin[0] repo default chunk size is just 200 tokens

this is anyway a limitation, no doubt, but chunking is pretty often used

[0] https://github.com/openai/chatgpt-retrieval-plugin/blob/main...

heya, supabase engineer here, i coauthored the part about benchmarking different models with pgvector to compare QPS when using fewer dims models.

btw, you can find the dataset with embeddings generated by all 3 mentioned: text-embedding-ada-002, all-MiniLM-L6-v2, and GTE-small on huggingface[0]

and big thanks to Stephan Sturges for his dataset[1]. we just extended his OpenAI ones and texts themselves with oss ones

[0] https://huggingface.co/datasets/Supabase/wikipedia-en-embedd...

[1] https://www.kaggle.com/datasets/stephanst/wikipedia-simple-o...

it means the precision at K(10) or precision for first 10 documents retrieved. In other words - number of relevant documents amongst retrieved 10(k) of them

So 0.98 precision would mean that 9.8 out of 10 documents are relevant on average

i am not Paul of course, but i just want to highlight that the idea behind blogpost was to show that if you set some settings for postgres correctly, you can achieve much better performance

ie you can notice that most of results are on the same hardware as in nirantk, and we achieved 7 times better RPS with 0.98 precision on it and 19 times better RPS with 0.91 precision on the same amount of cpu and memory

goal was to compare pgvector with pgvector rather than any other technology

supabase engineer here,

would be happy to share details regarding our research and outcomes if you have any questions

is it the right convo branch? as this is a chatgpt plugin and uses openai embeddings api, it for certain sends some data to their servers. both when embeddings are generated and when chatgpt sends request to your plugin and receives response back.

Hey Ahoha, thanks for caring about me. This is me who added myself as an author to the blog, i feel like “engineering” is the best description for the the work I do in Supabase :) You can find it in PR https://github.com/supabase/supabase/pull/14520

As for the work behind, copple made significant edits and additions (as I thought of them) to my draft, so i decided add him as coauthor (just to make sure: I wasn’t asked by anyone and copple only noticed this after publication)

heya, yep, if you add ivfflat index for embedding <column> in your table. I have to note that I am not pgvector contributor and this is how i understood it from the repo and its code. you can find a bit more context from the paper https://dl.acm.org/doi/pdf/10.1145/3318464.3386131 and https://github.com/pgvector/pgvector repo itself

Let me know if you want to play around with it and plug it in your ChatGPT, we can expose it if there is a need

hey hn, supabase engineer here

I just added a PR to the great retrieval plugin template repo from OpenAI to add Datastore options for Supabase and Postgres. This is an implementation of Datastore interface using Postgres database and 2 different interfaces: PostgREST for using with the ability to hook everything with your existing app and RLS policies, or just a plain database connection. They are completely swappable, the only thing you need to change is a couple of environment variables to switch from Postgres to Supabase and vice versa.

You will find an example of a plugin with full PostgresDB documentation embedded into postgres database in the post. And i will be happy to answer your questions here :)

Hey, from what we’ve seen up to 50k per node should be ok for supavisor on 4 cores as well. And TPS is here mainly to demonstrate that we not only create dummy connections but that these connections are really used. And 20k TPS is just the maximum amount that database node was able to handle. To bump tps we had to spin up more db replicas. We want to run these tests as well at some point, but now were just verifying that we can connect millions of clients and with some tangible tps but not overcomplicating the setup too much.

Hi, i was running these million connections benchmarks, happy to answer if you have any questions regarding this:) In short, that was pretty easy to do, only things we had to tweak to make it work were some elixir and linux limits

I like C4 for system design overview and sequence diagrams to show detailed process of interactions in one particular scenario. I found these two the most useful for me.

I also like to write diagrams as code with https://plantuml.com/sequence-diagram cause you can then add it to git and easily version it. Also, a lot of tools on the market have support for PlantUML