Ask HN: What tools do you use for data munging and merging?

Re:Ask HN: What tools do you use for data munging and merging? by wodenokoto on Monday January 13, 2020 @10:25AM (#22035190)

I really like R / Tidyverse, but that requires that you have more memory than your datasize (especially if you are using narrow and long dataframes, such as recommended by tidyverse) and it also requires you to code, instead of using a UI

[ Parent | on HN ]

Re:Ask HN: What tools do you use for data munging and merging? by stuxnet79 on Monday January 13, 2020 @11:02AM (#22035565)

Is there really no way of processing dataframes by chunks in the R ecosystem? I'm firmly in the Python camp but I wonder how much I'm missing out on in the R world.

[ Parent | on HN ]

Re:Ask HN: What tools do you use for data munging and merging? by deadcaribou on Monday January 13, 2020 @11:23AM (#22035752)

There are many ways to process larger than RAM objects in R.

The dbplyr package makes it possible to use a local db table as if they are in-memory data frames (https://dbplyr.tidyverse.org/).

https://diskframe.com/ is also a pretty great solution that processes data by chunks.

[ Parent | on HN ]

Re:Ask HN: What tools do you use for data munging and merging? by onearmedscissor on Monday January 13, 2020 @11:33AM (#22035868)

Take a look at data.table https://cran.r-project.org/web/packages/data.table/

[ Parent | on HN ]

Re:Ask HN: What tools do you use for data munging and merging? by livingmargot on Monday January 13, 2020 @07:17PM (#22040351)

Sorry to butt in, but could you point me to a resource on processing data in chunks using Python?

[ Parent | on HN ]