News for hackers. Stuff that matters.
Re:Show HN: rqlite v2.2.2, replicating SQLite via Raft conse... by tyingq on Tuesday April 26, 2016 @11:16PM (#11577620)
Well done. I am curious, though, if you tried implementing this as a sqlite VFS. That would allow it to be more plug-n-play with existing software that already talks to sqlite.

I suppose you would have had to hack up something to map offset ranges into key/value pairs, but at a high level, it seems feasible.

[ Parent | on HN ]
Re:Show HN: rqlite v2.2.2, replicating SQLite via Raft conse... by sargun on Wednesday April 27, 2016 @01:53AM (#11578095)
I built one recently. We have a SQLite VFS built on top of Zookeeper for doing metadata-like stuff. We can pull between 1000-100000 reads per seconds out of it.

I realize that this is a total abuse of what Zookeeper does, and how it is supposed to work, but hey, this is a solution that's _simple_ and _correct_.

Here are the benchmarks comparing the traditional VFS with the ZK VFS: https://gist.github.com/sargun/7503ba636c1a0eb2e2f5cda7a6a23...

PS: Feel free to ask me any questions

[ Parent | on HN ]
Re:Show HN: rqlite v2.2.2, replicating SQLite via Raft conse... by tyingq on Wednesday April 27, 2016 @07:02AM (#11579311)
That's the sort of thing I was interested in. Did you split the data up into kv pairs somewhat like this shim: http://www.sqlite.org/src/doc/trunk/src/test_multiplex.c ?

Or is just a small database where 1 sqlite file == 1 key?

[ Parent | on HN ]
Re:Show HN: rqlite v2.2.2, replicating SQLite via Raft conse... by otoolep on Wednesday April 27, 2016 @02:06AM (#11578145)
Pretty interesting. Do you have source code you can share?
[ Parent | on HN ]
Re:Show HN: rqlite v2.2.2, replicating SQLite via Raft conse... by sargun on Wednesday April 27, 2016 @02:57AM (#11578342)
Not yet open source, and it probably wont be for a while, given it's in $EMPLOYER's proprietary product.
[ Parent | on HN ]
Re:Show HN: rqlite v2.2.2, replicating SQLite via Raft conse... by otoolep on Tuesday April 26, 2016 @11:27PM (#11577646)
I looked up SQLite VFS. That's pretty interesting, though I am not sure if rqlite has sufficient control over that low a layer in the system, due to the way rqlite is built.

I opened an issue anyway, to look into this.

https://github.com/rqlite/rqlite/issues/108

[ Parent | on HN ]
Re:Show HN: rqlite v2.2.2, replicating SQLite via Raft conse... by tyingq on Tuesday April 26, 2016 @11:35PM (#11577677)
I was really just curious if you had looked at it and ruled it out. Pretty sure it would be a different approach not compatible with the way rqlite works. At the VFS level, you're only seeing the reads/writes/locks to the data file...no visibility to the sql.
[ Parent | on HN ]
Re:Show HN: rqlite v2.2.2, replicating SQLite via Raft conse... by sargun on Wednesday April 27, 2016 @01:55AM (#11578103)
If you implement a VFS, you have to either implement the WAL subsystem yourself, which isn't pluggable at the moment, or go with journaling mode, which is kinda meh.
[ Parent | on HN ]
Re:Show HN: rqlite v2.2.2, replicating SQLite via Raft conse... by tyingq on Wednesday April 27, 2016 @07:25AM (#11579401)
>>If you implement a VFS, you have to either implement the WAL subsystem yourself

It's pluggable in the sense that it maps the the shared memory calls for WAL to your calls. But yeah, figuring out how to emulate shared memory across a cluster is on the implementer.

[ Parent | on HN ]
Re:Show HN: rqlite v2.2.2, replicating SQLite via Raft conse... by zmmmmm on Tuesday April 26, 2016 @11:46PM (#11577710)
Is this a potential way to work around the single-writer nature of SQLite? It's probably the main limitation I hit when using SQLite, which otherwise fits a lot of jobs very well. Obviously you would not want to use it for situations where there is contention, but SQLite locks the whole database even if you are writing to completely separate tables.
[ Parent | on HN ]
Re:Show HN: rqlite v2.2.2, replicating SQLite via Raft conse... by otoolep on Tuesday April 26, 2016 @11:49PM (#11577721)
No, unfortunately not, as every change must go through a single node i.e. the leader. The same limitations as a standalone SQLite database apply.
[ Parent | on HN ]
Re:Show HN: rqlite v2.2.2, replicating SQLite via Raft conse... by ganitayu on Tuesday April 26, 2016 @10:07PM (#11577344)
rqlite ensures that every change made to the database is made to a quorum of SQLite files, or none at all.

So that means if a single node in the cluster fails then everything stops?

[ Parent | on HN ]
Re:Show HN: rqlite v2.2.2, replicating SQLite via Raft conse... by otoolep on Tuesday April 26, 2016 @10:12PM (#11577362)
No. As long as (n/2)+1 nodes are up, where n is the size of the cluster, everything continues to work.
[ Parent | on HN ]
Re:Show HN: rqlite v2.2.2, replicating SQLite via Raft conse... by asimuvPR on Tuesday April 26, 2016 @10:44PM (#11577512)
Stupid question: What if I want the databases to be built with optional extensions (like the json extension)? Is there a way to deal with such scenario?

Cool project!

[ Parent | on HN ]
Re:Show HN: rqlite v2.2.2, replicating SQLite via Raft conse... by otoolep on Tuesday April 26, 2016 @11:24PM (#11577639)
I'm not really sure I follow you. But if it requires compiling other stuff into the source code, probably not. The SQLite source is built along with the SQLite Go layer, without that control AFAIK.
[ Parent | on HN ]
Re:Show HN: rqlite v2.2.2, replicating SQLite via Raft conse... by asimuvPR on Wednesday April 27, 2016 @12:32AM (#11577832)
Ok, that answers it perfectly. Thank you!
[ Parent | on HN ]
Re:Show HN: rqlite v2.2.2, replicating SQLite via Raft conse... by alexnewman on Tuesday April 26, 2016 @11:27PM (#11577650)
I don't think now works. Great idea though! Great demonstration ifnot nothing else. Can we do one over 2pc as well?
[ Parent | on HN ]
Re:Show HN: rqlite v2.2.2, replicating SQLite via Raft conse... by otoolep on Tuesday April 26, 2016 @11:30PM (#11577662)
Yeah, certain non-deterministic functionality can't be used. This is called out here:

https://github.com/rqlite/rqlite#limitations

Though with now() you could calculate that before you issue the query.

[ Parent | on HN ]
Re:Show HN: rqlite v2.2.2, replicating SQLite via Raft conse... by FatAmericanDev on Tuesday April 26, 2016 @08:25PM (#11576952)
This is cool. On a related topics, I'm still waiting for a sqlite for graphs to emerge.
[ Parent | on HN ]