How does SQLite work? Part 2: btrees

Re:How does SQLite work? Part 2: btrees by fnord123 on Sunday June 28, 2020 @01:25PM (#23670380)

Let's get all the records from 2020-06-28T10:00:00.000Z to 2020-06-29T09:59:59.999Z and sum the quantity field. Or anywhere you would want to group_by. Or anywhere you might use something like an RDD that partitions the data to compute something for each chunk of data (in parallel) (also after a group_by).

Also, sparse vectors. You don't have a value to store for each index so you use a BTreeMap. I've done this using sentinel values in an array, but that's gross depending on how sparse and how large the vector is.

In Python I would just use a Dataframe (pandas), but in Rust I would use BTreeMap. I wouldn't wish for all languages to have a dataframe type in them since they're such complicated beasts with tricky APIs (pandas and R are such acquired tastes) so I think people can get quite far with B(+)Trees. So that's what I wish was in every stdlib.

[ Parent | on HN ]

Re:How does SQLite work? Part 2: btrees by BiteCode_dev on Sunday June 28, 2020 @05:45PM (#23672324)

I don't know, wouldn't dumping the data into a db easier? If it fits on sqlite, you already have a solution in most languages to this problem.

Technically you would be using a btree, but a higher level interface, like with a fs or hashmap. Which is kinda my point.

I dont say it's not a good DS, but having it in all languages stdlib seems overkill given the tool we have today.

I'm a python guy, and even without panda it has the heapq module which is a b-tree, yet I never really got a reason to use it.

[ Parent | on HN ]

Re:How does SQLite work? Part 2: btrees by fnord123 on Monday June 29, 2020 @02:08AM (#23674837)

Heapq is a priority queue implemented on a binary tree; not a btree.

> I dont say it's not a good DS, but having it in all languages stdlib seems overkill given the tool we have today.

C++ has map and Java has TreeMap. These are both red black trees (which are equivalent to btrees where k=4). So there's no question of whether there are ordered mapping types in these languages. IMO they'd be better as b(+)rees with a wider fanout. That's all.

[ Parent | on HN ]