News for hackers. Stuff that matters.
Re:Big Book of R by dajtxx on Thursday April 10, 2025 @09:12PM (#43649470)
I am working on a system at present where the data scientist has done the calculations in an R script. We agreed upon an input data.frame and an output csv as our 'interface'.

I added the SQL query to the top of the R script to generate the input data.frame and my Python code reads the output CSV to do subsequent processing and storage into Django models.

I use a subprocess running Rscript to run the script.

It's not elegant but it is simple. This part of the system only has to run daily so efficiency isn't a big deal.

Re:Big Book of R by shoemakersteve on Thursday April 10, 2025 @10:51PM (#43650051)
Any reason you're using CSV instead of parquet?
Re:Big Book of R by epistasis on Friday April 11, 2025 @12:56AM (#43650618)
CSV seems to be a natural and easy fit. What advantage could parquet bring that would outweigh the disadvantage of adding two new dependencies? (One in Python and one in R)
Re:Big Book of R by pjacotg on Friday April 11, 2025 @01:06AM (#43650664)
Not the op, but I started using parquet instead of CSV because the types of the columns are preserved. At one point I was caching data to CSV but when you load the CSV again the types of certain columns like datetimes had to be set again.

I guess you'll need to decide whether this is a big enough issue to warrant the new dependencies.

Re:Big Book of R by pletnes on Friday April 11, 2025 @02:12AM (#43650957)
Many of the reasons csv is bad is because you don’t control both reader and writer. Here, if you’re 2 persons that collaborate OK, they should be fine.