Re:Numba: High-Performance Python with CUDA Acceleration
by j88439h84 on Thursday September 21, 2017 @01:44PM
(#15305806)
Are you saying if that you have missing data you can't use numba, or if you have missing data, and you use numba together with pandas, that pandas will handle the missing data where numba alone could not?
Re:Numba: High-Performance Python with CUDA Acceleration
by grej on Thursday September 21, 2017 @02:10PM
(#15306014)
Heavy numba user here. What Wes is saying is that while Pandas handles some of those missing values in an automated way, if you choose to use numba it uses numpy arrays so you may have to handle some of those things yourself. I have at times used a separate numpy array to indicate whether values are missing or not. You could also use a value which is far out of the bounds of what you might ever see in your real data, then test for that while you're looping over those values (eg. fill missing values with -3.4E38 if you have a float32).
Depending on what you're doing, you might be able to use numpy.nan as a value. It does work inside of numpy arrays. But some methods that operate on those objects might not work as you expect.
For instance, if you run numpy.mean on a numpy array of [nan, 4, 5], it will return nan. If you run the same thing on a pandas dataframe of the same values, you'll get 4.5.
pandas creator here. Numba is a complementary technology to pandas, so you can and should use them together. It is designed for use with NumPy arrays and so does not deal with missing data and other things that pandas does. It does not help as much with non-numeric data types.