News for hackers. Stuff that matters.
Re:Apache Arrow 3.0 by jzer0cool on Wednesday February 03, 2021 @08:33PM (#26021533)
Could you expand on this more columnar data and row data? I missed something here how the data is organized and what you mean by the C,D values getting dragged along.
Re:Apache Arrow 3.0 by repsilat on Thursday February 04, 2021 @12:19AM (#26022869)
To rephrase the sibling comment, if you had an array of four {ABCD} structs, there's basically two ways of storing them on disk:

1. AAAA BBBB CCCC DDDD

2. ABCD ABCD ABCD ABCD

One major heuristic in how CPUs make your code fast is to assume that if you access some memory, you're probably interested in the memory nearby. So when you access the first "A" bit of memory (common to both sequences above), depending on the memory layout you use, the CPU might also be smart and load the next bits into memory too -- maybe the next "AA", maybe "BC".

Depending on your workload, one or the other of those might be faster. If you're only interested in the first ABCD element because you're doing

  SELECT * FROM users WHERE id=$1
then you'll likely want "row-oriented" data -- the #2 scheme above. But if you're interested in all of the A values and none of the values from B/C/D because you're doing

  SELECT AVG(age) FROM users
then you'll likely want something "column-oriented" -- the #1 scheme above.
Re:Apache Arrow 3.0 by int_19h on Wednesday February 03, 2021 @10:37PM (#26022303)
Row data is an array of structs, with each struct representing a record.

Columnar data is a struct of arrays, with each array representing a column.