I/O is no longer the bottleneck

Re:I/O is no longer the bottleneck by avinassh on Saturday November 26, 2022 @07:59AM (#33752269)

I am working on a project [0] to generate 1 billion rows in SQLite under a minute and inserted 100M rows inserts in 33 seconds. First, I generate the rows and insert them in an in-memory database, then flush them to the disk at the end. To flush it to disk it takes only 2 seconds, so 99% of the time is being spent generating and adding rows to the in-memory B Tree.

For Python optimisation, have you tried PyPy? I ran my same code (zero changes) using PyPy, and I got 3.5x better speed.

I published my findings here [1].

[0] - https://github.com/avinassh/fast-sqlite3-inserts

[1] - https://avi.im/blag/2021/fast-sqlite-inserts/

[ Parent | on HN ]

Re:I/O is no longer the bottleneck by iamchp on Monday November 28, 2022 @06:56AM (#33772503)

Thank you for this (albeit a different project). My immediate question when I read the last part of the page was 'What about PyPy!?' PyPy might be different for the original post's execution, but I still assume some significant speedups?

[ Parent | on HN ]

Re:I/O is no longer the bottleneck by avinassh on Tuesday November 29, 2022 @09:28AM (#33787979)

> PyPy might be different for the original post's execution, but I still assume some significant speedups?

Yes, I believe so as well. They have done many CPU optimisations, so it is likely to be faster

[ Parent | on HN ]