News for hackers. Stuff that matters.

A comment on SQLite: 35% Faster Than the Filesystem. Parent comment

Re:SQLite: 35% Faster Than the Filesystem by breck on Sunday January 15, 2023 @04:50AM (#34388180)
Yes I did an experiment years ago with 6.5 million files and it was disaster (https://breckyunits.com/building-a-treebase-with-6.5-million...).

However, things have totally changed with the M1 generation of Macbooks. Things that were once near impossible now run in an instant. I need to redo this experiment.

[ Parent | on HN ]
Re:SQLite: 35% Faster Than the Filesystem by _dhruva on Sunday January 15, 2023 @10:12AM (#34389962)
When I worked at NetApp, this was a problem there too.

IIRC, the fix was for directory entries were later stored in sorted buckets (aka hash map) and only the corresponding bucket for the file name would get locked. This reduced scope of lock for atomic operations like rename and also allowed faster lookup instead of O(n) based scan.

[ Parent | on HN ]