News for hackers. Stuff that matters.
Re:Tank – A very high performance distributed log service by biokoda on Thursday December 01, 2016 @12:30AM (#13077399)
No performance numbers. No tests. Nothing about how it manages concurrency, how it actually writes to disk, how it does replication.

It appears to be a server not a lib. Why would someone use this instead of kafka?

Re:Tank – A very high performance distributed log service by markpapadakis on Thursday December 01, 2016 @02:42AM (#13077902)
It is a service, and a library(C++). You can always check the implementation, but obviously that's the wrong answer to that question :)

The README states that replication is not implemented yet. I encourage you to also check this issue https://github.com/phaistos-networks/TANK/issues/14 for some details on the I/O semantics, and the Wiki for answers to other questions: https://github.com/phaistos-networks/TANK/wiki

Re:Tank – A very high performance distributed log service by biokoda on Thursday December 01, 2016 @03:09AM (#13077994)
So single threaded, blocking file io. Not very impressive or useful (as a lib).
Re:Tank – A very high performance distributed log service by markpapadakis on Thursday December 01, 2016 @03:22AM (#13078024)
Yes, single threaded, because the contention to the various files and the cost of serialisation would likely negate the benefits of using multiple threads.

Network I/O is obviously asynchronous - you may want to check the codebase. Disk I/O is synchronous, but: - uses sendfile() and read readahead() to reduce or eliminate the likelihood of blocking reads: see here please https://github.com/phaistos-networks/TANK/issues/14#issuecom...

- AIO is either broken or supported on XFS (depending on kernel release, and also in the past, appending to a file on an XFS fs could block and degrade performance). But not using DirectI/O so writes end up in memory and only get flushed if it hits the commit count, or periodically - so, particularly for local disks, HDDs or SSDs, this in practice never blocks for more than a few ms when flushing, if at all.

Re:Tank – A very high performance distributed log service by biokoda on Thursday December 01, 2016 @03:33AM (#13078059)
> Yes, single threaded, because the contention to the various files and the cost of serialisation would likely negate the benefits of using multiple threads.

Multithreaded does not mean multiple files.

> But not using DirectI/O so writes end up in memory and only get flushed if it hits the commit count, or periodically - so, particularly for local disks, HDDs or SSDs, this in practice never blocks for more than a few ms when flushing, if at all.

Unless you are actually using it under a high throughput scenario which is why you would use a lib like this. It will work great until you hit the actual flush point, then possibly block for seconds even minutes.

If your performance needs are high, mandating XFS is not unreasonable.

Re:Tank – A very high performance distributed log service by krenoten on Thursday December 01, 2016 @10:25AM (#13079925)
It's totally unreasonable to mandate a particular FS for modern general purpose software.

Multiple files does mean multiple threads. Unless you manage to run this without a modern operating system.

Re:Tank – A very high performance distributed log service by biokoda on Thursday December 01, 2016 @12:12PM (#13081111)
> It's totally unreasonable to mandate a particular FS for modern general purpose software.

That is completely debatable. We are talking about a "a high performance distributed log service". Not a word processor.

> Multiple files does mean multiple threads.

Which is something entirely different from what I said...

Re:Tank – A very high performance distributed log service by the_duke on Thursday December 01, 2016 @04:54PM (#13083896)
You are making all kinds of assumptions without knowing anything about the implementation.
Re:Tank – A very high performance distributed log service by jitl on Thursday December 01, 2016 @01:31AM (#13077625)
They address this in the README by saying you should use Kafka.
Re:Tank – A very high performance distributed log service by biokoda on Thursday December 01, 2016 @01:50AM (#13077689)
They say tank is a good choice of you need simplicity and performance. Yet they do not provide any evidence of it being faster than Kafka or any indication of it being fast at all.

If you're not mentioning kernel calls, allocations, thread synchronization, io mechanism etc. you're not going to be very believable.

Re:Tank – A very high performance distributed log service by markpapadakis on Thursday December 01, 2016 @02:43AM (#13077906)
Please see earlier comment. Specifically re: sys calls and concurrency you may want to start from this issue https://github.com/phaistos-networks/TANK/issues/14
Re:Tank – A very high performance distributed log service by mfukar on Thursday December 01, 2016 @06:33AM (#13078716)
Starting from that, you may not want to dismiss AIO & O_DIRECT in the span of a paragraph.