News for hackers. Stuff that matters.
Re:Noms – A versioned, forkable, syncable database by aboodman on Sunday October 16, 2016 @01:12AM (#12717082)
Hi Hacker News. I'm one of the founders of the Noms project and Attic Labs, the company behind it. Happy to answer any questions.

In the meantime, as long as I've got your attention, here's a few new stuffs we've been working on since last time Noms was discussed here in August:

- A prototype query language, and a demo of how to create indexes in Noms: https://www.youtube.com/watch?v=fv6_T5yaWns

- Support for merging concurrent (and potentially conflicting) changes: https://www.youtube.com/watch?v=--7dgoJBdjU

Re:Noms – A versioned, forkable, syncable database by mostafah on Sunday October 16, 2016 @02:49AM (#12717356)
Does/will Noms have proper support for multiuser sync? That’s an issue with CouchDB.

I mean: instead of syncing whole the database, only syncing the parts that a user has access to, and being able to define those accesses. The standard use case for consumer apps.

Re:Noms – A versioned, forkable, syncable database by aboodman on Sunday October 16, 2016 @11:29AM (#12719024)
Noms syncs at the level of a "value" not the entire database. Since a Noms database is a tree of values, you can sync the entire db, but you don't have to. You can also sync a single list.

So yes, in principle, we can definitely do this. In practice we are missing some conveniences that would make it a really easy drop-in feature.

Re:Noms – A versioned, forkable, syncable database by rwmj on Sunday October 16, 2016 @06:51AM (#12718023)
There's quite a dramatic claim on the website, "merge [...] changes efficiently and correctly days, weeks, or years later." How does that work? For example if you have two records saying userid 3's name is "ann" and userid 3's name is "jane", I don't see how you could merge those without extra information or human input.
Re:Noms – A versioned, forkable, syncable database by aboodman on Sunday October 16, 2016 @11:19AM (#12718970)
The claim on the website is not meant to suggest that any two changes can be automatically merged. I will try to clarify that.

The world contains logical conflicts because physical constraints mean that processes can operate disconnected from each other. No database can wave that away.

Noms will automatically, efficiently, and correctly merge changes that don't logically conflict. Which is a pretty cool and unique property in a database.

If any conflicts are found, there is a callback to user software to perform a resolution.

More info in the documentation:

https://godoc.org/github.com/attic-labs/noms/go/merge

Re:Noms – A versioned, forkable, syncable database by nradov on Sunday October 16, 2016 @11:57AM (#12719180)
IBM Domino (aka Lotus Notes) has been automatically, efficiently, and correctly merging changes that don't logically conflict since 1989. How is the functionality in Noms unique?
Re:Noms – A versioned, forkable, syncable database by aardvark179 on Sunday October 16, 2016 @08:05AM (#12718239)
It's a very hard problem to solve. The best you can really manage is to automatically merge non-conflicting changes, and defining what is a conflicting change will require some extra knowledge if you have multiple tables with relations between them.

For extra fun try doing this with geographic data and try merging geometry changes correctly.

Re:Noms – A versioned, forkable, syncable database by striking on Sunday October 16, 2016 @09:49AM (#12718563)
Well, hopefully you're not just using an incrementing user id field. Twitter's Snowflake format is much closer to "definitely unique". So then you have a very simple set, which you could very simply merge with other sets. So that might help you get started with merging those pieces of data.

But I agree, there are certain kinds of data that you can't so efficiently merge. Your best bet is to try to adapt it into some sort of known, proven CRDT. (Dunno how you do that with this database, haven't really read up on it.)

Re:Noms – A versioned, forkable, syncable database by dman on Sunday October 16, 2016 @08:28AM (#12718295)
Any pointers to design docs or papers that inspired Noms?
Re:Noms – A versioned, forkable, syncable database by ComodoHacker on Sunday October 16, 2016 @06:29AM (#12717942)
What about ACID? How far do you plan to divert from it?
Re:Noms – A versioned, forkable, syncable database by senand on Sunday October 16, 2016 @08:28AM (#12718294)
More specifically, what isolation guarantee do you make? Are writes linearizable? What consistency guarantees do you claim? How well do you think you will do when reviewed by aphyr (https://aphyr.com)?
Re:Noms – A versioned, forkable, syncable database by aboodman on Sunday October 16, 2016 @08:57PM (#12722023)
You should understand that Noms is doing something quite a lot simpler than the systems that are usually reviewed on Aphyr.

Noms doesn't manage its own storage - it relies on an underlying key/value store that must provide strongly consistent reads for at least one key. In other words, we delegate most of the hard part to somebody else.

With that all said...

Currently our intent is that:

- Transactions that read and write from a single dataset have strong serializability

- Transactions that read from multiple datasets and write to a single dataset have snapshot isolation

- Transactions that write to multiple datasets aren't possible

In the future, we will probably allow additional configuration, such that, e.g., one could choose snapshot isolation within a dataset for additional concurrency, or strong serializability for transactions that span datasets.

Re:Noms – A versioned, forkable, syncable database by botw on Sunday October 16, 2016 @09:54PM (#12722210)
curious on how the irregular black shape title bar is created?
Re:Noms – A versioned, forkable, syncable database by Normal_gaussian on Sunday October 16, 2016 @07:32AM (#12718151)
This is the most important question in my mind.

If it isn't ACID it needs to make a very strong case for itself to even be played with by most DBA's, including myself.

Re:Noms – A versioned, forkable, syncable database by Ericson2314 on Sunday October 16, 2016 @09:43AM (#12718547)
Depending on details, content addressing could make it very hard to not be ACID. Here's to hoping.
Re:Noms – A versioned, forkable, syncable database by aboodman on Sunday October 16, 2016 @09:09PM (#12722060)
Yes, it kind of falls out of content-addressing. See: https://news.ycombinator.com/item?id=12722023
Re:Noms – A versioned, forkable, syncable database by amelius on Sunday October 16, 2016 @07:53AM (#12718205)
How do you deal with access control? I can imagine that it would be necessary in many applications to have parts of the database only accessible to some people, and that access rules could be complicated and also depend on values in the database itself.
Re:Noms – A versioned, forkable, syncable database by aboodman on Sunday October 16, 2016 @11:34AM (#12719049)
We haven't implemented access control yet. See https://github.com/attic-labs/noms/issues/1183 for one idea.
Re:Noms – A versioned, forkable, syncable database by esfandia on Sunday October 16, 2016 @05:03AM (#12717725)
Looks really interesting! This may sound like a stupid question, but do you have any publication I can cite by any chance?
Re:Noms – A versioned, forkable, syncable database by chuhnk on Sunday October 16, 2016 @08:15AM (#12718265)
What was the process like for raising funding for an OSS project? Especially one that was pre-launch.
Re:Noms – A versioned, forkable, syncable database by seego on Sunday October 16, 2016 @08:28AM (#12718297)
Do you have any plans to provide C/C++ bindings?
Re:Noms – A versioned, forkable, syncable database by aboodman on Sunday October 16, 2016 @11:51AM (#12719145)
We want to. It would be nice if we could just use cgo but it's not complete. So we need to build some kind of simplified API to Noms that can be exported via cgo.

I created a bug for this just now: https://github.com/attic-labs/noms/issues/2718

Please feel free to get involved there.

Re:Noms – A versioned, forkable, syncable database by freekh on Sunday October 16, 2016 @05:17AM (#12717759)
Wow, congrats - this looks really interesting!