Writing a SQLite clone from scratch in C

Re:Writing a SQLite clone from scratch in C by convolvatron on Monday September 04, 2017 @11:15AM (#15168780)

i think that topic is larger than the presentation thus far. hopefully memory mapped flash will obviate the need for alot of this but:

   o you need to make an on-disk structure (i.e. btree) to let you search your tables (indices)
   o that has to be fronted by an in memory cache
   o your on-disk structure should maintain consistency even if the power fails and some writes get lost
     - often but not always this is facilitated by keeping a separate write-optimized structure called a write-ahead-log
     - if you use a WAL, you'll also need to implement the replay mechanisms to get the primary indices back up to date on a failure
   o because the disk operations are expensive and high latency you'll need to start managing concurrency explicitly

back ends are alot of work. unfortunately this is a place where the lack of decent concurrency mechanisms in your language and OS interface can really cause alot of headache.

pedagogically, i guess you would start with a completely synchronous non fault tolerant btree? or a maybe just the log, and introduce trees as a read accelerator?

[ Parent | on HN ]

Re:Writing a SQLite clone from scratch in C by noam87 on Monday September 04, 2017 @01:52PM (#15169749)

What are some good readings / places to start on this topic?

[ Parent | on HN ]

Re:Writing a SQLite clone from scratch in C by convolvatron on Monday September 04, 2017 @02:32PM (#15169999)

thats a really good question, unfortunately everything I learned was by working with people who knew more than I did. its pretty sad that alot of systems work is that way.

this book looks to be pretty comprehensive, and explicitly discusses on-disk structures, write ahead logging, and recovery.

  http://infolab.stanford.edu/~ullman/dscb.html

relational languages and transactions get alot more playin the academy..i think its because you can make sense of them in some abstract way without getting sucked into involved discussions about caching heuristics and OS write guarentees, scheduling and fussy performance characteristics

[ Parent | on HN ]

Re:Writing a SQLite clone from scratch in C by jwhitlark on Monday September 04, 2017 @05:34PM (#15170997)

I like http://dataintensive.net/, though it's a slightly different focus.

[ Parent | on HN ]