- replication is in the works, this is going to be both TCP and UDP based, column-first, very fast.
- yes, benchmarks are indeed are done on second pass over the mmaped pages. First pass would trigger IO, which is OS-driven and dependant on disk speed. We've seen well over 1.5Gb/s on disks that support this speed. Columns are mapped into memory separately and they are lazy accessed. So the memory footprint depends on what data your SQLs actually lift. We go quite far to minimize false disk reads by working with rowids as much and possible. For example 'order by' will need memory for 8 x row_count bytes in most cases.
- durability is something we want user to have control over. Under the hood we have these commit modes:
https://github.com/questdb/questdb/blob/master/core/src/main...
NOSYNC = means OS flushes memory whenever. That said, we use sliding 16MB memory window when writing. Flushes will trigger by unmapping pages. ASYNC = we call msync(async) SYNC = we call msync(sync)
I was also going to ask about replication. Any idea when it's going to be done?
Oh and kudos for the witty (previous) company name: Appsicle, haha, love that.
(1) Is it a single-server only, or is it possible to store data replicated as well?
(2) I’m guessing that all the benchmarks were done with all the hot data paged into memory (correct?); what’s the performance once you hit the disk? How much memory do you recommend running with?
(3) How’s the durability? How often do you write to disk? How do you take backups? Do you support streaming backups? How fast/slow/big are snapshot backups?