brynb
“be the change” or some such
to address the part of your comment that doesn't make my head spin: only very occasionally do i see senior Go developers import 3rd party libraries. i'm just speaking from my experience
NB: there’s that whole discourse about “how are you all multi-millionaires when the pay for senator/representative is at most a few hundred thousand a year?”
(FYI, that’s a burner email— I’ll reply from a different one)
Turns out it makes for a great self-hosted git solution, so I made a git remote plugin that allows you to interact with it using regular vanilla git commands: https://github.com/redwood/redwood/tree/libp2p-connectivity2...
https://github.com/brynbellomy/redwood
Redwood is a decentralized database providing a data model that's much like a Redux state tree. One of the fundamental design goals was to better harmonize how data is handled on frontend and backend layers.
It's also:
- syncable/offline-capable, thanks to a CRDT system
- able to serve as a standalone Git backend
- embeddable into Go applications
- immutable, in a blockchain-y sense, due to its append-only ledger
There are some demos in the Github repo:
- A real-time collaborative text editor
- A chat application
- Git integration
- A video streaming application (still improving performance here, but it's functional)
It's pretty alpha still, but feedback and contributions from anyone interested would be very welcome.
- Redwood (https://github.com/brynbellomy/redwood), a realtime, p2p database. Data is structured in state trees that evolve over time. Merge algorithms are configurable at any point in a given state tree, but the default is a CRDT algorithm called "sync9".
- Braid (https://github.com/braid-work and https://braid.news), a draft IETF spec that specifies certain extensions to HTTP that make it much easier to build systems like Redwood. The Braid spec is under active development on Github, and we welcome input from anyone interested in the idea.
- Axon (http://axon.science and https://github.com/AxonNetwork), marketed as a platform for making it easier to collaborate on scientific research, but under the hood, it's basically just some extensions to git that allow you push commits over a peer-to-peer network bound together by a DHT.
I would also highly recommend getting involved in the Internet Archive's DWeb meetups and conferences, where you'll find hundreds of people interested in solving exactly these kinds of problems.
For what it's worth, I have a silly side project goal of reimplementing Bitcoin on top of Redwood, just to prove it can provide the same (general) security guarantees. Not sure I'll get around to that any time soon, though :)
Anyway, here's what we do:
1. Users are identified by a public/private keypair (actually using Ethereum's implementation at the moment, just so I could focus on other stuff). All transactions must be signed by the sender, allowing recipients to verify the sender identities.
2. You can place "transaction validators" at any node in your state trees, and any transaction affecting the subtree under the validator will be checked by that validator. Currently, there's a "permissions" validator included that gives a simple way to control writes based on the Ethereum keypair I mentioned above. The current implementation of the permissions validator isn't very well fleshed out yet -- it just lets you specify a true/false write permission associated with a [user, keypath regex] pair -- but it's a decent placeholder for now, until I can get around to roles and policies.
3. You can also write custom transaction validators in Go/Lua/Javascript, which should make it trivial to implement just about any access control model you desire.
4. You can also create a "private" tree by explicitly specifying the set of users who are allowed to read from and write to that tree. The default Redwood node implementation does automatic peer discovery and keeps a list of peers whose identities/addresses have been verified. When it receives a transaction for a private tree, it only gossips that transaction to the tree's members (as opposed to its behavior with public trees, which is to gossip transactions to any peer who subscribes to that tree).
5. We also want secure persistent storage for transactions so that we can get high availability and redundancy without compromising the security model. To facilitate this, we allow you to configure the node to talk to what we're calling a "remote blind store" (essentially a key-value DB running on a separate piece of hardware, possibly off-site). The Redwood node encrypts transactions with a key the remote store doesn't possess and sends it over gRPC to the blind store.
6. Some applications will have a high volume of transaction data, and will want to be able to prune/merge/rebase that data to save space. To prevent bad actors from issuing malicious prune requests, the remote blind stores are going to implement their own p2p protocol involving threshold signatures. Prune requests will only be honored if a quorum of these blind stores sign off on them.
We've been thinking about this quite a bit, but not publicly. I'd love for someone to help us punch holes in it!
Redwood might be better described as a "p2p application platform" than a realtime database, but there's plenty of overlap. It's written in Go, supports HTTPS, Libp2p, and WebRTC transports (and soon, a Cap'n Proto RPC interface). It can also:
- serve as a Git remote
- stand in as an application server (you can upload assets directly to it, and they're served over regular HTTPS)
- execute Lambda-like functions written in Go, Lua, and Javascript that can be installed at any point in a given state tree
Would love some early feedback, and also to discuss similarities and differences with Sapphire's model. I've set up some demos in the "scenarios" folder -- a chat app, a realtime document editor, and a Git integration. Take a look if you're interested:
https://github.com/brynbellomy/redwood
So far, the model seems promising. It's fully peer-to-peer, and supports decentralized identity, configurable conflict resolution, read/write access, asset storage, and currently is running across 3 different transports:
- HTTP (augmented with extensions proposed by the Braid project [1][2])
- Libp2p
- WebRTC
I've included two simple demos, a collaborative document editor (well, it's just a textarea at the moment), and a chat app. Would appreciate any feedback or participation that folks are willing to give.
The basic idea is distributed version control, like git, but over p2p swarms rather than clusters around “central” repositories. We have special handling for large datasets (but still using git) to improve transfer efficiency and diffing.
There’s a UI layer for collaboration (discussion, PRs, review) that supports deep linking to and embedding of files at specific commits, which sounds a bit like what you’re looking for.
Feedback is very much appreciated!
EDIT: so far, it draws from Aikido, Dialectical Behavior Therapy, the psychology of video games and virtual experiences — new additions will continue to be as diverse in origin (hopefully).