News for hackers. Stuff that matters.
Re:Doing a database join with CSV files by atombender on Wednesday January 01, 2020 @01:02AM (#21926873)
PostgreSQL can do it on live data without importing anything. Set up the "file_fwd" extension:

  CREATE EXTENSION file_fdw;
  CREATE SERVER fdw FOREIGN DATA WRAPPER file_fdw;
Declare tables:

  CREATE FOREIGN TABLE person (id int, sex text)
    SERVER fdw
    OPTIONS (filename '/tmp/person.csv', format 'csv');
  CREATE FOREIGN TABLE weight (id int, weight int)
    SERVER fdw
    OPTIONS (filename '/tmp/weight.csv', format 'csv');
Now:

  SELECT * FROM person JOIN weight ON weight.id = person.id;
Re:Doing a database join with CSV files by haddr on Wednesday January 01, 2020 @07:29AM (#21927781)
In R this is rather straightforward, and you can also achieve it without additional libraries:

  weight <- read.csv("weight.csv")
  person <- read.csv("person.csv")
  merge(weight, person, all=TRUE)
Of course, nowadays you would use data.table, but still the merging logic would be exactly the same.
Re:Doing a database join with CSV files by atombender on Wednesday January 01, 2020 @03:52PM (#21930546)
That loads all the data into memory, though. Fine if you're joining all of it, but with the Postgres example you can throw the whole SQL language at the data, and it will be streamed and pipelined efficiently just like with normal Postgres tables.
Re:Doing a database join with CSV files by nkozyra on Wednesday January 01, 2020 @07:52AM (#21927838)
I never got deep into R as we had python or R as options in school for data projects so forgive me but where is the database join in this scenario?
Re:Doing a database join with CSV files by haddr on Wednesday January 01, 2020 @08:53AM (#21927998)
Original article doesn't say anything about "database join". It's about joining two datasets by some common ID.

R in this case fits the bill and even allows for some relational algebra here (e.g. INNER JOIN would be merge(X, Y, all=FALSE), LEFT OUTER JOIN: merge(X, Y, all.x=TRUE), etc...)

Re:Doing a database join with CSV files by stilisstuk on Wednesday January 01, 2020 @08:53AM (#21927999)
Merge joins the two dataset on identical columns. All=True is an outer join.
Re:Doing a database join with CSV files by kragen on Thursday January 02, 2020 @01:27PM (#21938126)
This isn't "without importing anything". This is "importing all the data again every time you run a query".
Re:Doing a database join with CSV files by atombender on Thursday January 02, 2020 @01:34PM (#21938190)
No, "import" means "create state from data". It's not creating any persistent state at all.

But it's an important nuance for another reason, because you can operate on the original files in real time. You can update the underlying files and your queries will reflect the changes.

The downside is since the data is streamed at query time, it's less efficient and doesn't allow indexes to be created.

Re:Doing a database join with CSV files by kragen on Thursday January 02, 2020 @01:50PM (#21938386)
That's reasonable.