News for hackers. Stuff that matters.

A comment on Facebook open-sources LogDevice, a distributed storage for sequential data.

Re:Facebook open-sources LogDevice, a distributed storage fo... by akavel on Thursday September 13, 2018 @02:02AM (#17976394)
Can someone from FB chime in with some info how much storage is needed for the logs/data? Say, for 1 GB of raw input logs from a http server (nginx/apache), when stored in LogDevice would they take notably less space on disk (compression), or more (overhead)? This interests ne for evaluating resources/costs I'd need to prepare if I were to deploy it...
[ Parent | on HN ]
Re:Facebook open-sources LogDevice, a distributed storage fo... by cedricvg on Thursday September 13, 2018 @05:28AM (#17977211)
These numbers really depend on the compressibility of the content, compression scheme and the type of batching used. The metadata overhead is fairly minimal. LogDevice allows you to configure this on either the client, sequencer or rocksdb level.
[ Parent | on HN ]
Re:Facebook open-sources LogDevice, a distributed storage fo... by akavel on Thursday September 13, 2018 @07:53AM (#17977856)
Is some form of compression enabled by default, without having to tweak options?
[ Parent | on HN ]
Re:Facebook open-sources LogDevice, a distributed storage fo... by sh00s on Thursday September 13, 2018 @02:47PM (#17981242)
No, compression is disabled by default. You can enable compression either on the storage layer, or by enabling batching and compression on the sequencer, or by using the buffered write API with compression on the client.
[ Parent | on HN ]
Re:Facebook open-sources LogDevice, a distributed storage fo... by SirMonkey on Thursday September 13, 2018 @06:23AM (#17977441)
have you had a look at https://github.com/oklog/oklog https://www.youtube.com/watch?v=gWWK2eyZ-sc

I think it's fairly simple and might be enough. Can't comment on storage requirements thou.

[ Parent | on HN ]
Re:Facebook open-sources LogDevice, a distributed storage fo... by are595 on Thursday September 13, 2018 @08:28AM (#17978023)
Dang, I was looking at oklog earlier, but looks like it is archived now...
[ Parent | on HN ]