Re:Show HN: Dropbase 2.0 – Turn offline files into live data...
by mikorym on Tuesday August 18, 2020 @02:23PM
(#24202245)
I've used Pandas recently, not sure if this will ever help you, but dictionaries are much faster if you continually add rows.
pandas.concat and similar functions for appending to a Pandas dataframe can be quite slow. Just mentioning this in case you ever encounter this issue; maybe you don't need to. In my case I changed to dictionaries for important parts and 2 mins changed to 5 seconds execution time.
However, in my case, I have to change logical row structure and not just read in rows as is.
pandas.concat and similar functions for appending to a Pandas dataframe can be quite slow. Just mentioning this in case you ever encounter this issue; maybe you don't need to. In my case I changed to dictionaries for important parts and 2 mins changed to 5 seconds execution time.
However, in my case, I have to change logical row structure and not just read in rows as is.