Now, I think you could definitely set up a system that could handle the distributed data like this and allow you to access it, but I am skeptical that you could construct such a system that would be able to read/write/update/delete files on the fly and retain N-node failure tolerance at all times. You would probably have to allow the system to replicate data slowly.
There is also the problem of network splits. When they rejoin, how do they resolve conflicts and such?
<node_name>_<year>-<month>-<day>_<hr>-<min>-<sec>.tar.xz
into the filesystem and another server would be reading through everything and processing it (not in real time, but as a batch system).It's ok for things to be left over for collection at the next cycle.
Also netsplits aren't much of an issue if you're largely just doing writes and reads on seperate files. It only becomes a problem for short-term and currently processed files (that are being read and written to in real time).
This is more of a long-term cold storage system and not nodes talking to each other.
TL;DR
"It's not a bug it's a feature"
Patient: "Doctor, how do I stop the pain I get from doing this"
Doctor: "You stop doing that"You set up one introducer, where storage nodes and clients meet and exchange each other's address. After that your clients can use the storage as one big single, end-to-end encrypted, resilient space where N nodes out of M can fail without any effect on your data.
Edit: Also, can you say "this group of nodes count's as a single location" for failure protection? So if I have two locations where I store servers and then 10 other one off data collection locations can I say "I want you to treat these datacenters as one node since they are very likely to fail together if they fail".
Here it is directly from the Q&A...
" Q12: If I had 3 locations each with 5 storage nodes, could I configure the grid to ensure a file is written to each location so that I could handle all servers at a particular location going down? "
" A: Not directly. We have a wiki page and some tickets (linked from the wiki page) about this but it's deeper than it looks and we haven't come to a conclusion on how to build it.
The current system will try to distribute the shares as widely as possible, using a different pseudo-random permutation for each file, but it is completely unaware of server properties like "location". If you have more free servers than shares, it will only put one share on any given server, but you might wind up with more shares in one location than the others.
For example, if you have 15 servers in three locations A:1/2/3/4/5, B:6/7/8/9/10, C:11/12/13/14/15, and use the default 3-of-10 encoding, your worst case is winding up with shares on 1/2/3/4/5/6/7/8/9/10, and not use location C at all. The most likely case is that you'll wind up with 3 or 4 shares in each location, but there's nothing in the system to enforce that: it's just shuffling all the servers into a ring, starting at 0, and assigning shares to servers around and around the ring until all the shares have a home.
The possible distributions of shares into locations (A, B, C) are:
(3, 3, 4) 1500 (2, 4, 4) 750 (2, 3, 5) 600 (1, 4, 5) 150 (0, 5, 5) 3 sum = 3003
So you've got a 50% chance of the ideal distribution, and a 1/1000 chance of the worst-case distribution. "
But RAID and Syncthing aren't substitutes for backups, for the same reason: no restore if your data get hosed.
RW times would be slower but storage would be bigger and more redundant.
Basically redundant but distributed storage with N node fail tolerance. Preferably interfaceable via a "/mnt/cloud" directory so I can use it from my laptop and easily write backup scripts and store data into it.
If this existed I'd have something big to work on setting up next week.