Re:Engineers Shouldn’t Write ETL
by jjoonathan on Friday March 18, 2016 @04:43PM
(#11314833)
Reasons to prefer sed/awk: you're in bash
Reasons to prefer sqlite: it's easier to embed in an app, you want sort, join, split-apply-combine, scale, transactions, compression, etc.
Reasons to prefer pgsql: sqlite's perf tools suck compared to pgsql (last time I got stuck anyway) and I'm sure there are lots of sql-isms that sqlite doesn't handle if that's your jam. EDIT: forgot everything-is-a-string in sqlite, just wanted to add that it has bit me before.
Another reason to use Unix tools like sed and awk (and grep and sort and...) is that, if you just need to iterate over all your data and don't need a random-access index, they are really really fast.
don't forget bdb!
No. If you are already comfortable in PgSQL, there is no need to use SQLite.
You will see PostgreSQL will come in handy once you get beyond the initial import stage..
PostgreSQL's type system will come to your aid . SQlite essentially treats everything as string, which can turn nasty when you get serious with your queries etc.,
question: is SQLite incrementally helpful when I'm already comfortable with a local pgsql db to handle the use case you suggested? would SQLite be redundant for me in this case?
question: between postgres and unix tools (sed, awk) is there reason to use SQLite?