emerson_clarke
First there are stupid people, and then there are stupid people who know they are stupid, then there are smart people who think they are smart, and finally there are smart people who know they are actually not that smart.
Self awareness is the differentiating factor at every level.
Intelligence comes in many forms; practical, creative, mathematical, logical, social, emotional, etc. As members of society we are more or less forced to specialise and must accept that being really good at something usually means we are terrible at something else. So if you are half as smart as you think you are, you will learn to recognise all the areas where you really are not, which vastly outnumber the ways in which you are.
Which is a bit like saying: "I dont know half of you half as well as I should like; and I like less than half of you half as well as you deserve."
I for one am not that smart, im just a lot less stupid than most ;)
However, if you need to know the count/size...
Then in general you must accept that the count/size of the write stack is dynamic, so even if you use an integer to track the value (and keep updated with atomic exchange or increment), at the point that you read its value on one thread it may have already changed on another.
So it doesn't really matter if you reset the value to 0 using an interlocked exchange (this cant be done as an atomic unit with respect to the head pointers unless your platform supports a double CAS). Some loss of count/size information will occur.
Alternatively you can safely iterate through the stack to calculate the count/size anytime you need it (provided the next pointers are also treated atomically).
Neither option will give you the exact size since as discussed above its always dynamic, but this i no way impedes the functioning of the write/flush stacks as described. In my solution i just set the value of the count/size to 0, and disregard any loss of information since it is not critical.
Information on lock free data structures and the caveats (they are extremely difficult to get right) is freely available though:
http://www.cs.cmu.edu/~410-s05/lectures/L31_LockFree.pdf
http://en.wikipedia.org/wiki/ABA_problemFor example, on an LP64 architecure:
Item * first = writers.first;
while( CAS((long *)&writers.first,(long)0,*((long*)&first)) != *((long*)&first))
{
first = writers.first;
}
User threads write log lines directly to buffers from an allocator usually via a TLS mediated stream. The use of an allocator avoids locking on system calls during memory allocation and minimizes copying between user code and eventual flush to disc/network. Buffers are written to the write stack using atomic CAS, if no buffers are available from the allocator the user thread may spin, or force a flush in the same way as the flushing thread.
A single flushing thread watches the write stack on a timer and when it reaches some threshold it uses an atomic CAS to switch the head pointer between the flush and write stacks before enumerating the flush stack and writing all buffers to disc/network freeing the buffers back to the allocator (again using atomic operations). It is subtle but if done right user threads and the flushing thread interact optimally in response to demand.
This solution is much more flexible than a ring buffer and also much simpler and faster in testing than any complicated patterns like disruptor and competitive with expensive hardware logging solutions.
It recognizes the fact that much of the overhead in logging comes from expensive copying of log data in memory, and it also ensures that minimum context switching takes place which is essential if you are not to defeat the entire point of fast lock free algorithms.
Unlike a ring bufffer it has no blocking or performance degredation when the buffer gets full and requires no large chunk of memory to be permanently allocated, although the allocator may periodically allocate new temporary memory if its buckets are full or if a log line is too large for a the maximum bucket size.
What you end up with is a logging system where user threads are minimally impacted during writes and throughput is able to max out the disc/network.
If you want to know how C++ can be better, you should take a look at at the Reason C++ framwork, especially if you have never seen C++ in its most pure uncomplicated OO form.
Reason is a C++ framework that ive been writing for about 8 years, which aleviates many of the sources of pain highlighted in this discussion. Writing code with Reason is much more like using a dynamic language, with a full library like Java or .Net.
It supports raii, but doesnt use exception handling and doesnt bother with const nonsense or over the top inheritance restrictions. In fact pretty much everything in Reason is public and designed to be derived, modified or enhanced.
It has generic programming features, but doesnt force you to write everything as templates (a classic mistake that the standard library makes). For example, you can have an iterator of int's without caring what the underlying container type is, so your code and algorithms can actually be generic, not just infectious templates.
Unlike a lot of C++ frameworks and libraries, its not specifically focused on networking, or a few esoteric template classes or collection libraries.
Reason is a fully featured systems programming framework with a complete interface to all the usual posix api's and everything else you would expect from a modern language. It has strings, regexes, streams, parsers, sockets, threads, filesystem, encryption, encoding, xml, xpath, collections, time and date, sql (mysql, postgress, sqlite), http (client/server), smart pointers, formatting, logging, and much more.
But the features are perhaps not as important as how it is written. It is simple, and very object oriented, everything is designed to work togeather cleanly and obey the principle of least surprise. Just like Python and Ruby, Reason allows you to do a lot in a very small amount of code.
So if your wondering how you can have the simplicity of C, with just the good parts of C++, heres your answer.
For those with a slightly broader perspective and an inquisitive mind, perhaps you should take a look at at the Reason C++ framwork, specially if you have never seen C++ in its most pure uncomplicated OO form.
Reason is a C++ framework that ive been writing for about 8 years, which aleviates many of the sources of pain highlighted in this discussion. Writing code with Reason is much more like using a dynamic language, with a full library like Java or .Net.
It supports raii, but doesnt use exception handling and doesnt bother with const nonsense or over the top inheritance restrictions. In fact pretty much everything in Reason is public and designed to be derived, modified or enhanced.
It has generic programming features, but doesnt force you to write everything as templates (a classic mistake that the standard library makes). For example, you can have an iterator of int's without caring what the underlying container type is, so your code and algorithms can actually be generic, not just infectious templates.
Unlike a lot of C++ frameworks and libraries, its not specifically focused on networking, or a few esoteric template classes or collection libraries.
Reason is a fully featured systems programming framework with a complete interface to all the usual posix api's and everything else you would expect from a modern language. It has strings, regexes, streams, parsers, sockets, threads, filesystem, encryption, encoding, xml, xpath, collections, time and date, sql (mysql, postgress, sqlite), http (client/server), smart pointers, formatting, logging, and much more.
But the features are perhaps not as important as how it is written. It is simple, and very object oriented, everything is designed to work togeather cleanly and obey the principle of least surprise. Just like Python and Ruby, Reason allows you to do a lot in a very small amount of code.
So if your wondering how you can have the simplicity of C, with just the good parts of C++, heres your answer.