I have used SQLite databases sharded at 2GB due to mobile fs limitations, storing 10-14GB in total, total lookup times across whole query was still on the order of 100ms untuned, unoptimized.

SQLite has your back.