There were lots of pre-v1.0 design discussions on this vs sentinels, such as https://discourse.julialang.org/t/representing-nullable-valu... which can be dug up. The conclusion was more about the safety than performance, since sentinel values are fundamentally unsafe in generally since they can take a valid value and make it "special" (like in the blog post, minimum integer). Thus Julia has the following 3 choices:
* Use `nothing`, which throws an error at each operation like `nothing + 1`. This requires `if x === nothing` handling everywhere, is the safest option, but is generally considered clunky for statistics use. As `nothing` is a singleton type, it has the same optimizations as what's mentioned in the blog post.
* Use `missing`, which propagates, i.e. `missing + 1 === missing`. This allows for any `Union{Missing,T}` to use a scheme similar to the blog post: as `nothing` is a singleton type, it has the same optimizations as what's mentioned in the blog post.
* Use a sentinel of your choice, risks known and dependent on the type.
I have run into issues where it would be nice to have some sort of missing type/class hierarchy though, in that the reason for missingness is coded somewhere and then how to represent that in those three categories gets complicated. At some level the different missing types get treated the same for many functions (such as your missing + 1 propagation example) but for other things not. Then you end up being tempted to use the third option which has its own issues. I suppose you could create another variable with type of missing information but then that also becomes complex quickly.
I'm being really loose and am unfamiliar with Arrow but it seems as if these kinds of discussions are useful but also maybe heavily language-dependent. E.g., how Julia, Python, R, Rust, or whatever interfaces with Arrow is also relevant.
Treatment of null and missing data in the context of data science has rarely gotten the full consideration it probably warrants in design.
Interesting read though.