The main benefit of a columnar representation in memory is it's more cache friendly for a typical analytical workload. For example, if I have a dataframe:

  (A int, B int, C int, D int)

And I write:

  A + B

In a columnar representation, all the As are next to each other, and all the Bs are next to each other, so the process of (A and B in memory) => (A and B in CPU registers) => (addition) => (A + B result back to memory) will be a lot more efficient.

In a row-oriented representation like protobuf, all your C and D values are going to get dragged into the CPU registers alongside the A and B values that you actually want.

Column-oriented representation is also more friendly to SIMD CPU instructions. You can still use SIMD with a row-oriented representation, but you have to use gather-scatter operations which makes the whole thing less efficient.

Could you expand on this more columnar data and row data? I missed something here how the data is organized and what you mean by the C,D values getting dragged along.

To rephrase the sibling comment, if you had an array of four {ABCD} structs, there's basically two ways of storing them on disk:

1. AAAA BBBB CCCC DDDD

2. ABCD ABCD ABCD ABCD

One major heuristic in how CPUs make your code fast is to assume that if you access some memory, you're probably interested in the memory nearby. So when you access the first "A" bit of memory (common to both sequences above), depending on the memory layout you use, the CPU might also be smart and load the next bits into memory too -- maybe the next "AA", maybe "BC".

Depending on your workload, one or the other of those might be faster. If you're only interested in the first ABCD element because you're doing

  SELECT * FROM users WHERE id=$1

then you'll likely want "row-oriented" data -- the #2 scheme above. But if you're interested in all of the A values and none of the values from B/C/D because you're doing

  SELECT AVG(age) FROM users

then you'll likely want something "column-oriented" -- the #1 scheme above.

Row data is an array of structs, with each struct representing a record.

Columnar data is a struct of arrays, with each array representing a column.

It might be a stretch , but isn't it kind of the same idea as why ECS is faster than OO in video games development ?

A columnar format is almost always what you want for analytical workloads, because their access patterns tend to iterate ranges of rows but select only a few columns at random.

About the only thing protocol buffers has in common is that it's a standardized binary format. The use case is largely non-overlapping, though. Protobuf is meant for transmitting monolithic datagrams, where the entire thing will be transmitted and then decoded as a monolithic blob. It's also, out of the box, not the best for efficiently transmitting highly repetitive data. Column-oriented formats cut down on some repetition of metadata, and also tend to be more compressible because similar data tends to get clumped together.

Coincidentally, Arrow's format for transmitting data over a network, Arrow Flight, uses protocol buffers as its messaging format. Though the payload is still blocks of column-oriented data, for efficiency.

This is intended for analytical workloads where you're often doing things that can benefit from vectorization (like SIMD). It's much faster to SUM(X) when all values of X are neatly laid out in-memory.

It also has the added benefit of eliminating serialization and deserialization of data between processes - a Python process can now write to memory which is read by a C++ process that's doing windowed aggregations, which are then written over the network to another Arrow compatible service that just copies the data as-is from the network into local memory and resumes working.

> It also has the added benefit of eliminating serialization and deserialization of data between processes

Is that accurate? It still has to deserialize from apache arrow format to whatever the cpu understands.

The important part to focus on is _between processes_.

Consider Spark and PySpark. The Python bits of Spark are in a sidecar process to the JVM running Spark. If you ask PySpark to create a DataFrame from Parquet data, it'll instruct the Java process to load the data. Its in-memory form will be Arrow. Now, if you want to manipulate that data in PySpark using Python-only libraries, prior to the adoption of Arrow it used to serialize and deserialize the data between processes on the same host. With Arrow, this process is simplified -- however, I'm not sure if it's simplified by exchanging bytes that don't require serialization/deserialization between the processes or by literally sharing memory between the processes. The docs do mention zero-copied shared memory.

The Arrow Feather format is an on-disk representation of Arrow memory. To read a Feather file, Arrow just copies it byte for byte from disk into memory. Or Arrow can memory-map a Feather file so you can operate on it without reading the whole file into memory.

That's exactly how I read every data format.

The advantage you describe is in the operations that can performed against the data. It would be nice to see what this API looks like and how it compares to flatbuffers / pq.

To help me understand this benefit, can you talk through what it's like to add 1 to each record and write it back to disk?

Redshift's explanation is pretty good. Among other things, if you only need a few columns there are entire blocks of data you don't need to touch at all. https://docs.aws.amazon.com/redshift/latest/dg/c_columnar_st...

It's truly magical when you scope down a SELECT to the columns you need and see a query go blazing fast. Or maybe I'm easily impressed.

- Selective/lazy access (e.g. I have 100 columns but I want to quickly pull out/query just 2 of them).

- Improved compression (e.g. a column of timestamps).

- Flexible schemas being easy to manage (e.g. adding more columns, or optional columns).

- Vectorization/SIMD-friendly.