Ask HN: Distributed File System
So I need some recommendations.
I've been building a distributed file system for work to store our hash tables with. These are 1 gig files (about 40TB worth of them) that are write once, read many.
It needs to duplicate the data across servers and make the file available via HTTP. Oh and it needs to scale quite well because as from next month we are potentially adding another TB per month.
So far I havent been able to find a DFS that does all the above so have been working on my own. But am nervous - the files are mission critical but I am not too worried about losing stuff per se (there are alternative backup solutions that make sure we have multiple static copies safe). Im more worried about not being able ot cope with the load. My current implementation is in Python and simply uses a central MYSQL server to track file locations.
So. Can anyone recommend a DFS option I have missed that fulfills my requirements. Or even better can anyone offer technical ideas to help with the development of our code.
:)
It doesn't serve via HTTP directly, but it's easy to point a web server at the filesystem (it has an Apache module to provide direct access without going through FUSE if you want higher performance).
http://www.gluster.org
The main concern is MYSQL is the single point of failure.
Don't make it a single point of failure, then. Set up several boxes. Heck, don't use MYSQL -- your update rate is going to be low (even if you have 10Gbps of incoming traffic, that's only one new GB file per second), so you could run the file-location service over DNS if you wanted.
SPOF concerns could be alleviated by having a standard master/slave failover setup. You might have other needs you haven't mentioned but otherwise sounds like MySQL can handle your needs for now, and keeping the setup as simple and off-the-shelf as possible is always a good thing ..
Do share more details of the setup you end up with eventually, though, it's an interesting topic and I for one will be curious to see what you came up with.
Hopefully some part of our system will be open sourced :)
I think there must be open source clones of the FS Google uses, but I don't know the names.
Cassandra looks pretty fun - are you suggesting that as the database right? Im thinking a quick python implementation for PUT (maybe DELETE) and meta operations, using cassandra as a backend and Lighttpd for the GET (high performance) might work..... cheers.
[1] http://www.nabble.com/Using-HDFS-to-serve-www-requests-td227...
[2] https://issues.apache.org/jira/browse/HADOOP-5010
Edit: OK, now I see that S3 was suggested but vetoed. Unwise decision, in my opinion. If you need data security, encrypt your data.
S3 is great for us atm but the work to encrypt it is too much because the cost doesnt scale for us long term. Potentially within a couple of years we will be spending 1/2 a million on bandwith and 1/2 a million on storage - which will continue upwards :)
Were facing the Google model: commodity hardware on a cheap T1 connection :)
I'm sure that it isn't all there yet, and your bug reports would help, but you might be interested in the overall approach -- applying the principles of test driven development (if it isn't automatically tested by your buildbot, then it isn't done) to packaging.
But like you I have also failed to get it to work on anything with any degree of success.
You mean that you tried to implement something on top of Tahoe, and it didn't work? Or Tahoe didn't work, so you went and wrote your own based on the Tahoe design? Or something else? :)
We have 4000 clients atm but hope to increase that quickly (20,000 within a year). That's TB's of data a month. SAN was considered but it too expensive to scale :( EDIT: well, not over the top expensive. But commodity servers with software is cheaper/better for us.
Also we at some point need a HTTP interface to the outside world.
If your access pattern allows for good caching you could also skip all the hassle of maintaining physical spindles and just move your stuff to S3. Their storage fees are quite affordable, 100T would set your back around $15k a month.
The killer with S3 is in the traffic bill, though. So that's only an option when most of your requests can be served from cache.