News for hackers. Stuff that matters.

A comment on Ask HN: What tools do you use for data munging and merging?. Parent comment

Re:Ask HN: What tools do you use for data munging and merging? by stuxnet79 on Monday January 13, 2020 @11:02AM (#22035565)
Is there really no way of processing dataframes by chunks in the R ecosystem? I'm firmly in the Python camp but I wonder how much I'm missing out on in the R world.
[ Parent | on HN ]
Re:Ask HN: What tools do you use for data munging and merging? by deadcaribou on Monday January 13, 2020 @11:23AM (#22035752)
There are many ways to process larger than RAM objects in R.

The dbplyr package makes it possible to use a local db table as if they are in-memory data frames (https://dbplyr.tidyverse.org/).

https://diskframe.com/ is also a pretty great solution that processes data by chunks.

[ Parent | on HN ]
Re:Ask HN: What tools do you use for data munging and merging? by onearmedscissor on Monday January 13, 2020 @11:33AM (#22035868)
Take a look at data.table https://cran.r-project.org/web/packages/data.table/
[ Parent | on HN ]
Re:Ask HN: What tools do you use for data munging and merging? by livingmargot on Monday January 13, 2020 @07:17PM (#22040351)
Sorry to butt in, but could you point me to a resource on processing data in chunks using Python?
[ Parent | on HN ]