SQLite is easily fast enough for what you propose, but if you're worried about it design a way to test it. Measure first, then optimize.

Plus, if you build it in a way that enables you to scale up (vertically to a 'better' database or horizontally to more SQLite databases) it won't even matter.