PandaPy has the speed of NumPy and the usability of Pandas

Re:PandaPy has the speed of NumPy and the usability of Pandas by Grimm1 on Friday January 24, 2020 @08:16PM (#22143983)

Pandas' type conversions irk me. I get why columns with NaN convert to float from integer but very rarely do I have data that is complete for every column and converting columns that were intentionally integer has caused headaches when that data then goes to other systems such as a sql db.

It is currently sitting at the center of an ETL system at my work (not my decision) and causes headaches.

[ Parent | on HN ]

Re:PandaPy has the speed of NumPy and the usability of Pandas by squaresmile on Friday January 24, 2020 @09:44PM (#22144287)

You can use "Int64" for nullable integer data type.

https://pandas.pydata.org/pandas-docs/stable/user_guide/inte...

[ Parent | on HN ]

Re:PandaPy has the speed of NumPy and the usability of Pandas by Grimm1 on Friday January 24, 2020 @10:50PM (#22144530)

Awesome super useful thank you.

[ Parent | on HN ]

Re:PandaPy has the speed of NumPy and the usability of Pandas by abakker on Friday January 24, 2020 @08:26PM (#22144024)

I mean, whenever I have this problem, I fall back on my (terrible) SPSS habits and just recode NaN to -999. With integer datasets that mostly works fine. You could come up with some alternative solution, too.

Personally, I'm an amateur at best, but Pandas has made it possible for me to do some really handy things.

Personal favorite feature: multindexing on rows AND Columns. Multiindex is such a common pattern, and so poorly handled in things like excel. Pandas really saves a lot of time with re-indexing, pivoting, stacking, or transposing data with multi indexes.

[ Parent | on HN ]

Re:PandaPy has the speed of NumPy and the usability of Pandas by qwhelan on Friday January 24, 2020 @08:57PM (#22144136)

You should upgrade your version of pandas if possible - that's been fixed for a few versions now.

[ Parent | on HN ]

Re:PandaPy has the speed of NumPy and the usability of Pandas by Grimm1 on Friday January 24, 2020 @11:01PM (#22144571)

Oh interesting I'll have to check, we did an upgrade pass a while ago maybe we just didn't upgrade pandas for whatever reason.

[ Parent | on HN ]

Re:PandaPy has the speed of NumPy and the usability of Pandas by qwhelan on Friday January 24, 2020 @11:29PM (#22144684)

As mentioned elsewhere in this thread, it's opt-in to avoid breaking existing behavior. But given that ingestion points are easy to identify, it's pretty straightforward to turn on (especially if you have a schema for your inputs): https://pandas.pydata.org/pandas-docs/stable/user_guide/inte...

[ Parent | on HN ]

Re:PandaPy has the speed of NumPy and the usability of Pandas by longemen3000 on Saturday January 25, 2020 @12:24AM (#22144890)

I saw in implementation (CSV parser in Julia) were the sentinel value was randomly assigned at read time (if a value in the input was equal to the sentinel value, change randomly).after parsing, the sentinel value would be converted to the appropriate data type (Julia Missing)

[ Parent | on HN ]

Re:PandaPy has the speed of NumPy and the usability of Pandas by Grimm1 on Saturday January 25, 2020 @01:04PM (#22147555)

That makes sense and thanks for the info and the link. It will be very useful going forward.

[ Parent | on HN ]