News for hackers. Stuff that matters.
Re:Batch size one billion: SQLite insert speedups, from the ... by ako on Tuesday September 26, 2023 @01:57PM (#37663251)
One alternative way I’ve been testing recently: load the entire csv file into a single row column in Postgres, and the use a view to parse it into lines and values. Then cache the result with a materialized view. 16mb, 500k rows (formula 1 laptimes) takes only a few seconds to load and map to materialized view records.
Re:Batch size one billion: SQLite insert speedups, from the ... by daza on Tuesday September 26, 2023 @04:07PM (#37665145)
What do you mean by "row column in Postgres"? It's confusing. Could you please clarify that?
Re:Batch size one billion: SQLite insert speedups, from the ... by Linosaurus on Tuesday September 26, 2023 @04:25PM (#37665345)
Sounds like the initial table has just one row and one column, containing the entire 16mb csv string.
Re:Batch size one billion: SQLite insert speedups, from the ... by ako on Tuesday September 26, 2023 @05:13PM (#37665891)
Correct, at least one row for every time you load a csv file.