Why HNSW is not the answer and disk-based alternatives might be more practical

Re:Why HNSW is not the answer and disk-based alternatives mi... by tw04 on Monday December 23, 2024 @03:52PM (#42497487)

> Its reliance on frequent random access patterns makes it incompatible with the sequential access nature of disk storage.

So use SSD? Are people seriously still trying to use spinning disk for database workloads in 2024?

[ Parent | on HN ]

Re:Why HNSW is not the answer and disk-based alternatives mi... by JoeAltmaier on Monday December 23, 2024 @03:53PM (#42497497)

Data farms are all about cost per GB. Spinning media is a fraction of the cost per.

[ Parent | on HN ]

Re:Why HNSW is not the answer and disk-based alternatives mi... by jandrewrogers on Monday December 23, 2024 @05:05PM (#42497984)

An SSD does not solve the problem of page fault chasing, it just makes it slightly less bad. This is fundamentally a software architecture problem.

This is solved with latency-hiding I/O schedulers, which don’t rely on cache locality for throughput.

[ Parent | on HN ]

Re:Why HNSW is not the answer and disk-based alternatives mi... by redskyluan on Monday December 23, 2024 @07:17PM (#42498633)

Even SSD won't be fast enough for most indexes due to the random access nature. I've seen more than 1M iops on a huge nvme disk when use DiskANN index

[ Parent | on HN ]

Re:Why HNSW is not the answer and disk-based alternatives mi... by VoVAllen on Monday December 23, 2024 @10:50PM (#42499608)

Hi, I'm the author of the article. The sequential access pattern of IVF makes prefetching and large block sequential reads much easier, whereas it's almost impossible for HNSW to achieve efficient prefetching.

[ Parent | on HN ]