News for hackers. Stuff that matters.
Re:SQLite 3.32 by mmsimanga on Saturday May 23, 2020 @09:11AM (#23282639)
I work in data analysis too and SQLite is to me what MS Excel is to other people in the industry. I have development background so I never picked up in MS Excel to same degree as my colleagues from statistics backgrounds. In corporate organisations where DBAs monitor the use of their database with an eagle eye SQLite is a fantastic option to analyse several millions of lines of data on your desktop. I find DB Browser for SQLite (DB4S)[0] is invaluable when using SQLite.

[0]https://sqlitebrowser.org/

Re:SQLite 3.32 by kyllo on Saturday May 23, 2020 @11:58AM (#23283978)
If you like SQLite for data analysis, you might want to check out DuckDB https://github.com/cwida/duckdb which is billed as "SQLite for analytics."

SQLite is a row store, which is best for OLTP (point queries, inserting/updating/deleting one or a few rows at a time), while DuckDB is a column store, which means the data layout has values from the same column stored contiguously, making aggregation queries (GROUP BY) perform much better.

Re:SQLite 3.32 by erichocean on Saturday May 23, 2020 @05:08PM (#23286467)
Wow, why isn't DuckDB not more widely known!?!?! Looks incredible!!!
Re:SQLite 3.32 by kyllo on Saturday May 23, 2020 @06:36PM (#23287197)
I was wondering the same--I guess because it's fairly new (2018) and it came out of a database research group at a European university, rather than a SV tech firm. Therefore, limited marketing budget.

By the way, here's a YT video of a talk given by one of the DuckDB implementers about why they made it, what it's for, and how it works: https://www.youtube.com/watch?v=PFUZlNQIndo

Re:SQLite 3.32 by nojvek on Sunday May 24, 2020 @12:35AM (#23289114)
SQLite is started by some Aussie devs right ?
Re:SQLite 3.32 by kyllo on Sunday May 24, 2020 @01:20AM (#23289325)
SQLite was started by Dwayne Richard Hipp from North Carolina. 20 years ago.
Re:SQLite 3.32 by justinclift on Sunday May 24, 2020 @03:57AM (#23289951)
> Aussie devs

Not SQLite itself. One of the DB Browser for SQLite dev's is an Aussie though. :)

Re:SQLite 3.32 by pachico on Saturday May 23, 2020 @01:44PM (#23284864)
This is great! Thanks for sharing!
Re:SQLite 3.32 by edwinyzh on Saturday May 23, 2020 @10:06AM (#23283027)
For Windows also check HeidiSQL (open source, very feature-rich)
Re:SQLite 3.32 by mmsimanga on Saturday May 23, 2020 @10:55AM (#23283434)
Thanks for this. I used to be a longtime user of HeidiSQL but drifted away because I landed up in an Oracle environment. At the time I drifted away SQLite was not supported. I look forward to being re-acquainted with HeidiSQL.
Re:SQLite 3.32 by iagovar on Saturday May 23, 2020 @09:35AM (#23282807)
Excel is nice as a simple sandbox, and PowerQuery is amazing IMO, the problem is that it's basically impossible to work with large datasets. If you didn't know about it I recommend you to play a bit with it.

And check SQLiteStudio for Windows, is nice too.

Re:SQLite 3.32 by nojito on Saturday May 23, 2020 @10:15AM (#23283106)
I handle hundreds of millions of rows in excel trivially.

PowerPivot is extremely amazing.

Re:SQLite 3.32 by kyllo on Saturday May 23, 2020 @11:49AM (#23283893)
Most people don't realize that Excel (like PowerBI) has an in-memory, compressed, column store database inside of it.

Loading hundreds of millions of rows into it takes a while, but given a commensurate amount of RAM and a reasonable data model (single table or star schema), performing aggregations with a pivot table is pretty snappy.

Re:SQLite 3.32 by speedgoose on Saturday May 23, 2020 @11:19AM (#23283622)
PowerBI is quite nice too.
Re:SQLite 3.32 by iagovar on Saturday May 23, 2020 @01:37PM (#23284808)
Well, for me with my personal PC with a quad core and 24gb DDR3, it gets unusable for about 1 million rows, but it's true that most of my datasets have around 10 columns or so.
Re:SQLite 3.32 by nojito on Saturday May 23, 2020 @01:42PM (#23284850)
You should be using PowerPivot in Excel.

Here's a video showing rapid pivots of 100m row dataset

https://youtu.be/0WwFJ0Zg3d8?t=2285

Re:SQLite 3.32 by iagovar on Saturday May 23, 2020 @02:04PM (#23285007)
Mmm, I was using PowerQuery and then use the editor to do the transformations (don't remember exact names sorry), and I just couldn't cope with it.
Re:SQLite 3.32 by mkl on Saturday May 23, 2020 @10:43AM (#23283349)
Does SQLiteStudio let you cancel an unexpectedly slow query part way through? DB Browser for SQLite doesn't seem to, and I don't really know what I'm doing so often accidentally set off a query that can't use indices without doing "EXPLAIN QUERY PLAN" first.
Re:SQLite 3.32 by justinclift on Sunday May 24, 2020 @03:48AM (#23289894)
> DB Browser for SQLite

If it helps, our development builds for quite a few months have:

https://nightlies.sqlitebrowser.org/latest/

Our recent 3.12.0-alpha1 release does as well:

https://nightlies.sqlitebrowser.org/latest/

The first beta release for 3.12.0 should be out next week. There's not much change in it though (mostly language string changes), as the alpha1 has turned out to be really stable. :)

Re:SQLite 3.32 by googie on Saturday May 23, 2020 @06:31PM (#23287155)
It does.
Re:SQLite 3.32 by giancarlostoro on Saturday May 23, 2020 @10:00AM (#23282991)
I used to use this but if you have a JetBrains sub you can open these files directly. Or even better if its a full subscription DataGrip is fantastic and they recently added MongoDB support. Which is not perfect but good enough!
Re:SQLite 3.32 by fauigerzigerk on Saturday May 23, 2020 @11:40AM (#23283818)
I have a lot of Excel-like use cases for which SQLite would be a great fit if only it had a decimal type.
Re:SQLite 3.32 by lifepillar on Sunday May 24, 2020 @02:18AM (#23289570)
I have developed SQLiteDecimal: http://chiselapp.com/user/lifepillar/repository/sqlite3decim...

Still very rough, but usable.

Re:SQLite 3.32 by justinclift on Sunday May 24, 2020 @03:50AM (#23289911)
> WARNING: this library is NOT ready for production. Use at your own risk!

How far off do you reckon it is from being "production" ready?

Asking because we've been adding useful SQLite extensions as optional extras in our (sqlitebrowser.org) installer. Can add yours too, if you reckon the code is reasonably cross platform and shouldn't cause (many) weird issues. :)

Re:SQLite 3.32 by cordite on Saturday May 23, 2020 @11:52AM (#23283926)
As in like Java Bigdecimal, with a BigInteger on the left side, and a fixed precision on the right?
Re:SQLite 3.32 by fauigerzigerk on Saturday May 23, 2020 @12:01PM (#23284000)
As in other database systems such as Postgres. Even a C# style decimal floating point type would be sufficient.
Re:SQLite 3.32 by kyllo on Saturday May 23, 2020 @06:32PM (#23287171)
Basically--it's a struct containing a byte array, weight, scale, and sign, and all the arithmetic operations are implemented in software. So it's really slow, and each RDBMS has to implement this type from scratch, or find a vendor lib that provides a suitable one, because C/C++ standard libraries don't provide one.
Re:SQLite 3.32 by darksaints on Saturday May 23, 2020 @03:26PM (#23285648)
It does have a decimal type.
Re:SQLite 3.32 by fauigerzigerk on Saturday May 23, 2020 @05:14PM (#23286519)
No, that's just an alias for numeric type affinity which results in a binary floating point type if there's a decimal point.

There's also a long tradition of "faking" decimals with integers. I have always found that to be extremely tedious and error prone.

Re:SQLite 3.32 by time0ut on Saturday May 23, 2020 @09:38AM (#23282830)
My tools of choice are similar: sqlite, DB browser, Jupyter. I've found that DB browser struggles with analytical queries on tables over a few GB though. Still an invaluable exploratory tool.
Re:SQLite 3.32 by iagovar on Saturday May 23, 2020 @09:44AM (#23282871)
Check SQLiteStudio. DBeaver also supports SQLite as far as I remember, but it's a bit more cumbersome and really geared towards mysql etc.