Re:Loading Data into Pandas: Tips and Tricks You May or May ...
by Helmut10001 on Thursday August 04, 2022 @10:47PM
(#32350968)
Not to taunt about the article, but the most important pandas parameters to me are `iterator=True` and `chunksize=x`, for streamed processing. Here's an example for processing a CSV file with 400 Million latitude and longitude coordinates.[1]
Re:Loading Data into Pandas: Tips and Tricks You May or May ...
by ashwal on Thursday August 04, 2022 @09:45PM
(#32350518)
Some good tips in here, I've find myself reaching for JSON/excel methods often.
Despite using it for years, I still haven't decided if pandas is poorly architected or if the clunkiness (for lack of better of term) is a result of the inherent difficulty of the tasks.
Re:Loading Data into Pandas: Tips and Tricks You May or May ...
by hansvm on Thursday August 04, 2022 @09:53PM
(#32350567)
I seem to remember the author awhile back writing about how it was their first major project and there were a ton of things they'd learned and would like to change.
Re:Loading Data into Pandas: Tips and Tricks You May or May ...
by bushbaba on Thursday August 04, 2022 @10:04PM
(#32350652)
I wish I knew about json_normalize sooner. JSON is great, but needing to transform it to a CSV/Excel sheet is a toil. Great to know about this one-liner!!
Re:Loading Data into Pandas: Tips and Tricks You May or May ...
by pineapplejuice on Thursday August 04, 2022 @11:51PM
(#32351370)
My tip is to keep a dict of all the fields and the data types you expect them to be, particularly strings. In my company we have IDs that start with zeros, or are a mix of numbers and letters, and get interpreted as numeric types. I'm frequently pulling data out of the DW with the same fields, so I just have to use the dtype= arg point it to my dict and it takes care of that for me.
[1]: https://ad.vgiscience.org/twitter-global-preview/00_Twitter_...
https://pandas.pydata.org/docs/reference/api/pandas.read_sql...
It’s much faster because DuckDB is vectorized. The result is a Pandas dataframe.
Querying the Pandas dataframe from DuckDB is faster than querying it with Pandas itself.
Despite using it for years, I still haven't decided if pandas is poorly architected or if the clunkiness (for lack of better of term) is a result of the inherent difficulty of the tasks.
https://github.com/capitalone/DataProfiler
The gist is that you can point to any common dataset and load it directly into pandas.
from dataprofiler import Data
data = Data("your_file.csv") # Auto-Detect & Load: CSV, AVRO, Parquet, JSON, Text, URL
I simply hate dealing with loading data, so it's my go-to.