News for hackers. Stuff that matters.
Re:Show HN: Pg_CRDT – an experimental CRDT extension for Pos... by kiwicopple on Saturday December 10, 2022 @07:43AM (#33932125)
For quick viewing, here's how it works in Postgres (using Yjs):

    create table posts (
      id serial primary key,
      content crdt.ydoc default crdt.new_ydoc()
    );
  
    insert into posts (content)
    values (crdt.new_ydoc());

    update posts 
    set content = content || crdt.new_ydoc()
    where id = 1;

The heavy-lifting is done by the teams at Yjs[0] and Automerge[1], who have re-implemented their JS libs in Rust. This made it possible for us to wrap them into a Postgres extension using pgx[2] (which is fast becoming a de-facto tool at supabase)

[0] Yjs: https://github.com/yjs/yjs

[1] Automerge: https://github.com/automerge/automerge

[3] pgx: https://github.com/tcdi/pgx

Re:Show HN: Pg_CRDT – an experimental CRDT extension for Pos... by Horusiath on Saturday December 10, 2022 @01:31PM (#33935267)
Yrs (Yjs on Rust) maintainer here: we actually had some idea about building extension to Postgres ;) Great to see that others share the excitement about this idea as well. See: https://github.com/y-crdt/y-crdt/issues/220
Re:Show HN: Pg_CRDT – an experimental CRDT extension for Pos... by kiwicopple on Saturday December 10, 2022 @01:53PM (#33935528)
perfect, we'd love to collaborate with you. I'll jump into your github after our Launch Week (this week), or feel free to reach out directly (contact details in my bio)
Re:Show HN: Pg_CRDT – an experimental CRDT extension for Pos... by cyanydeez on Monday December 12, 2022 @05:39PM (#33961480)
I really hope plv8 maintainers can finally ditch v8 And pave the way for using yjs in all tits glory.

Unfortunately, v8 basically is a bloated mess and postgres just stopped supporting it cause it's build process is horrendous.

Re:Show HN: Pg_CRDT – an experimental CRDT extension for Pos... by samwillis on Saturday December 10, 2022 @08:59AM (#33932662)
Super excited to see this Paul, and amused to see the nod to my comment in the post :-)

I'm going to enjoy getting stuck into this! (When I'm not stuck driving for six hours... just a quick coffee break reading this)

Support for CRDT columns within a db to enable JSON like indexing/querying capabilities is fundamental to then building some exciting further developments. Supporting both Yjs and Automerge is exactly the right route forward.

Re:Show HN: Pg_CRDT – an experimental CRDT extension for Pos... by samwillis on Saturday December 10, 2022 @12:06PM (#33934277)
Had a chance to take a little more of a look, did you investigate querying/indexing within the documents? (Didn't spot any sign in the repository)

Without the ability to index properties, or do ad-hock queries, on the documents there is little reason to to the merging in-DB. It then makes more sense to merge outside the DB and save both the raw CRDT object and a serialised JSON next to it for querying. I assume that if you had found it's performance better, indexing+querying would have been the next step?

However I'm not surprised you found performance not ideal, I suspect a layer in-front of PG throttling writes is the only way forward for a real-time use case.

Personally I find the offline + eventual consistency the most interesting use case. The ability to sync a users document set to a local db (SQLite or a future mini-pg/supabase) with local querying and then merge them back later. This could be as basic as a three column table (guid, doc, clock), the clock being a global Lamport like clock to enable syncing only changed documents.

Re:Show HN: Pg_CRDT – an experimental CRDT extension for Pos... by kiwicopple on Saturday December 10, 2022 @12:27PM (#33934493)
> This could be as basic as a three column table (guid, doc, clock), the clock being a global Lamport

This would be achievable, but also perhaps quite strict - we're dictating a table structure rather than working with existing CRDT libs. That said, we could probably do this today, without any mods/extensions

> querying/indexing

Yeah, there is nothing in this implementation for that

> Without the ability to index properties, or do ad-hock queries, on the documents there is little reason to to the merging in-DB

One benefit is the ability to send "directly" to the database, but I agree that lack of additional features in the current form outweighs this benefit

> a serialised JSON

This is how we might do it with with "Realtime as an authority" approach (and is the most likely approach tbh).

Re:Show HN: Pg_CRDT – an experimental CRDT extension for Pos... by aaronblohowiak on Saturday December 10, 2022 @12:16PM (#33934389)
Read modify write outside the db seems worse though?
Re:Show HN: Pg_CRDT – an experimental CRDT extension for Pos... by zombodb on Saturday December 10, 2022 @02:20PM (#33935820)
Always nice to see pgx being used in the wild! Awesome work.