Apache Arrow and the “Things I Hate About Pandas”

Re:Apache Arrow and the “Things I Hate About Pandas” by wesm on Tuesday September 26, 2017 @01:11AM (#15336356)

Right, but I don't think it's fair to say that R's support for categorical data is "excellent" if only strings can be category labels/levels. Categories (aka dictionary-encoded data in other systems) semantically may be any type in practice (strings, timestamps, numbers, etc.). The function as.factor in R is lossy because the input type is coerced to string.

[ Parent | on HN ]

Re:Apache Arrow and the “Things I Hate About Pandas” by rcthompson on Tuesday September 26, 2017 @03:07AM (#15336771)

I'm trying to think of an example where you would want to use a factor with levels that are timestamps rather than just a vector of timestamps. And likewise for other data types, even strings. The only case I can think of would be if you wanted to limit the possible values to specific set.

My impression is that factors in R are borderline-deprecated, especially in the tidyverse, in favor of just using the equivalent non-factor vector.

[ Parent | on HN ]

Re:Apache Arrow and the “Things I Hate About Pandas” by stewbrew on Tuesday September 26, 2017 @04:08AM (#15336996)

I think you're mixing up scales. Timestamps and numbers can be recoded to categories but when doing this you're throwing away information -- the same way you lose information when converting an integer to a string. When you work with factors you're no longer interested in qualities the data would have on its original scale like distance etc.

Could you please provide a real world example where this actually is a problem.

[ Parent | on HN ]

Re:Apache Arrow and the “Things I Hate About Pandas” by zzleeper on Tuesday September 26, 2017 @02:09AM (#15336558)

> the input type is coerced to string.

Wow. Didn't knew that.. strings is the one thing I always minimize in my datasets, due to speed and memory considerations..

BTW, I wanted to thank you for your 2012 slides on how you used hashes to group and join data. It led me to learn more about categoricals and I ended up implementing a Factor() object [1] in the other tool I use (Stata) that ended up being a life saver. In fact, once you have a powerful and fast categorical type, with a set of key functions, you can do anything from group the data, to count distinct categories, to run fixed effect regressions in no time.

[1] http://fmwww.bc.edu/repec/scon2017/Baltimore17_Correia.pdf

[ Parent | on HN ]

Re:Apache Arrow and the “Things I Hate About Pandas” by rcthompson on Tuesday September 26, 2017 @02:55AM (#15336722)

Memory isn't the issue. Both factors and string vectors only store each unique string once in memory. The issue is the limitation that factor levels can only be strings and not any other data type. Also, I think testing strings for equality is O(1) since it should simplify to a pointer comparison.

[ Parent | on HN ]

Re:Apache Arrow and the “Things I Hate About Pandas” by stewbrew on Tuesday September 26, 2017 @03:56AM (#15336953)

That's not right. Factor _levels_ can only be integers, factor _labels_ can only be strings since labels are the printable representation of a level. Maybe you could store the "real" values of a level as attribute (I.e. metadata) of the factor. Anyway, I think that's as solvable problem.

[ Parent | on HN ]

Re:Apache Arrow and the “Things I Hate About Pandas” by rcthompson on Tuesday September 26, 2017 @11:00AM (#15339232)

Well, I don't know if there's a standard terminology used elsewhere, but what you call labels, R calls levels.

[ Parent | on HN ]

Re:Apache Arrow and the “Things I Hate About Pandas” by stewbrew on Tuesday September 26, 2017 @11:30AM (#15339520)

Erm ... Well, I admit it's somewhat confusing.

    factor(x = character(), levels, labels = levels, exclude = NA, ordered = is.ordered(x), nmax = NA)

    [...]

    levels
    :   an optional vector of the values (as character strings) that x might 
        have taken. The default is the unique set of values taken by 
        as.character(x), sorted into increasing order of x. Note that this set 
        can be specified as smaller than sort(unique(x)).

    labels
    :   either an optional character vector of (unique) labels for the 
        levels (in the same order as levels after removing those in exclude), or 
        a character string of length 1.

This way, you can do something like that:

    > x <- 1:3
    > factor(x, levels = 1:2, labels = c("foo", "bar"))
    [1] foo  bar  <NA>
    Levels: foo bar

But this actually is:

    > factor(as.character(x), levels = c("1", "2"), labels = c("foo", "bar"))
    [1] foo  bar  <NA>
    Levels: foo bar

[ Parent | on HN ]