Re:PSA: SQLite WAL checksums fail silently and may lose data
by mardifoufs on Thursday July 24, 2025 @05:31PM
(#44676469)
I know MySQL has checksums by default, how does it compare? Is it useful or is it similarly weak?
Re:PSA: SQLite WAL checksums fail silently and may lose data
by jandrewrogers on Thursday July 24, 2025 @07:23PM
(#44677559)
I don't know but LLMs seem to think it uses a 32-bit CRC like e.g. Postgres.
In fairness, 32-bit CRCs were the standard 20+ years ago. That is why all the old software uses them and CPUs have hardware support for computing them. It is a legacy thing that just isn't a great choice in 2025.
Ext4 uses 16-/32-bit CRCs, which is very weak for storage integrity in 2025. Many popular filesystems for databases are similarly weak. Even if they have a strong option, the strong option is not enabled by default. In real-world Linux environments, the assumption that the filesystem has weak checksums usually true.
Postgres has (IIRC) 32-bit CRCs but they are not enabled by default. That is also much weaker than you would expect from a modern database. Open source databases do not have a good track record of providing robust corruption detection generally nor the filesystems they often run on. It is a systemic problem.
ZFS doesn't support features that high-performance database kernels use and is slow, particularly on high-performance storage. Postgres does not use any of those features, so it matters less if that is your database. XFS has traditionally been the preferred filesystem for databases on Linux and Ext4 will work. Increasingly, databases don't use external filesystems at all.