Re:Batch size one billion: SQLite insert speedups, from the ...
by ako on Tuesday September 26, 2023 @01:57PM
(#37663251)
One alternative way I’ve been testing recently: load the entire csv file into a single row column in Postgres, and the use a view to parse it into lines and values. Then cache the result with a materialized view. 16mb, 500k rows (formula 1 laptimes) takes only a few seconds to load and map to materialized view records.
[ Parent
| on HN ]
Re:Batch size one billion: SQLite insert speedups, from the ...
by daza on Tuesday September 26, 2023 @04:07PM
(#37665145)
What do you mean by "row column in Postgres"? It's confusing. Could you please clarify that?
Re:Batch size one billion: SQLite insert speedups, from the ...
by Linosaurus on Tuesday September 26, 2023 @04:25PM
(#37665345)
Sounds like the initial table has just one row and one column, containing the entire 16mb csv string.