Re:Factors are not first-class citizens in R
by mapcar on Wednesday September 24, 2014 @11:13AM
(#8361899)
Is rhdf5 the best library? I understand there is ncdf4 and h5r, and rgdal. I understood that R support for HDF5 was limited so I have not used R much for this purpose.
Re:Factors are not first-class citizens in R
by jzwinck on Wednesday September 24, 2014 @09:07PM
(#8365014)
h5r has been removed from CRAN "due to licensing problems"[1]. ncdf4 and rgdal use totally different formats. rhdf5 may not be the best library, but it seems to be the only library.
I don't think R support for HDF5 is "limited," I think it's just poorly implemented in a few ways (another is that you need a separate package to get 64-bit integer support, which is another problem with R in general).
Python's excellent h5py module and the official HDF5 tools like "h5dump" understand that a dataset is boolean if it has the type H5T_ENUM with two values "FALSE" (0) and "TRUE" (1).
But R works a different way: if you save a boolean vector from R to HDF5 (using the rhdf5 package), it will create a dataset of type H5T_STD_I32LE, which takes 4x the storage space. And if you read the boolean vector as understood by other tools, you will get factors. Then, the most amazing thing happens: R uses the string names of the enum ("FALSE" and "TRUE"), but remaps them to factors with 1 and 2. So if you do as.integer(h5read(...)) you will get a vector of ones and twos instead of zeros and ones. And of course R itself treats 1 and 2 as truthy values, so now your data is well and truly corrupted: all the elements are truthy!
That is: as.logical(as.integer(h5read(...))) always produces all TRUE values when the input is boolean written by a program other than R. This is a lovely source of bugs in real software.