Pgvector doesn't need to store everything in memory. It behaves similar to almost any Postgres AM's and store index data on disk. Performance-wise it's better to have enough memory for index data remain in shared memory buffers, but it is not a requirement for pgvector.