News for hackers. Stuff that matters.
Re:Many small queries are efficient in SQLite by imtringued on Tuesday February 16, 2021 @09:56AM (#26154392)
During the alpha Minecraft divided the world into 16x16x128 grids of blocks called chunks. Each chunk was its own file. Large worlds suffered from very poor performance because there were tens of thousands of files in a single folder.

Some random modder basically just put multiple chunks into one file so that each file is 2MB. If Notch had just put the game world into a SQLite database he wouldn't have had to reinvent the wheel. There are games that did that, such as the alpha of Cube World and they work just fine.

Heck, notch went one step further and invented NBT aka named binary tag which is basically a weirdo binary file format that stores JSON like data.

Re:Many small queries are efficient in SQLite by iforgotpassword on Tuesday February 16, 2021 @11:50AM (#26155888)
> Large worlds suffered from very poor performance because there were tens of thousands of files in a single folder.

It was using subdirs for the chunks, two levels iirc, one was chunkX % 36, the next level chunkY % 36. So there weren't that many files per directory. The slowness came from the overhead of opening, read/write and closing so many files all the time.

> Some random modder basically just put multiple chunks into one file so that each file is 2MB.

Almost, it wasn't limited by file size, it was putting 32*32 chunks into one file that was similar to a simple file system. The format of the individual chunks within that file stayed almost the same. Yet it performed much better.

NBT is indeed a little weird but fairly straight forward overall, I guess designing and implementing it just scratched an itch. It was a hobby project after all.

Re:Many small queries are efficient in SQLite by slaymaker1907 on Tuesday February 16, 2021 @05:06PM (#26159361)
I'm actually currently working on a user mode FS using Dokan for Windows that saves everything to a SQLite file for similar reasons. NTFS just doesn't do well at all with lots of small files.
Re:Many small queries are efficient in SQLite by fomine3 on Wednesday February 17, 2021 @12:22AM (#26163017)
Interesting. Did you tested any performance by non-NTFS?

I remembered this post. https://github.com/microsoft/WSL/issues/873#issuecomment-425...