Yes Sir, with this attitude I hope to make a remarkable progress in my tech career :) Seriously, there is nothing mysterious or magical about shoving a "plain-old JDBC call" into a Future.
http://en.wikipedia.org/wiki/Future_(programming)
Remarkable demo, btw. But lets not run down other approaches simply because one might, god forbid, have to "learn so many unfamiliar patterns".
You got it! Great job!
> Yes Sir, with this attitude I hope to make a remarkable progress in my tech career :)
Well one way to not make a remarkable progress in your career is use fads, acronyms, and unnecessarily complicated constructs. Why use futures in that example when actors perfectly model the problem domain? Are you showing off that you know about Futures and they are easy?
There are many ways to tackle concurrency, but IMO it's best to keep them separate as much as possible, or you quickly lose track of what's happening when.
I am also familiar with snobby wannabees functional programmers who instead of opening the goddam file and reading it are creating homomorphic endofunctors wrapped in futures with double memoization and distributed locks, so that nobody on the fucking team knows what's going on.
These people are 10x more dangerous than mediocre programmers who just find the simplest way to get the work done and ship the product.
Eventually 1% of the wannabes might get enlightened and realize that simple basic code is usually better than using every single programming concept wrapped in 100 lines of code that nobody (including themselves 2 weeks later) can understand.
Imo it's just another variation on the Yak Shaving theme with a dose of procrastination thrown in for good measure.
After a while the whole system broke down. Frankel wasn't paying any attention; he wasn't supervising anybody. The system was going very, very slowly - while he was sitting in a room figuring out how to make one tabulator automatically print arc-tangent X, and then it would start and it would print columns and then bitsi, bitsi, bitsi, and calculate the arc-tangent automatically by integrating as it went along and make a whole table in one operation.
Absolutely useless. We had tables of arc-tangents. But if you've ever worked with computers, you understand the disease - the delight in being able to see how much you can do. But he got the disease for the first time, the poor fellow who invented the thing.”
― Richard P. Feynman, Surely You're Joking, Mr. Feynman!
Exactly. And fascination with technology is important, it is what keeps people learning and searching, finding better tools. The problem with it, is it has a pathological side.
Like the tool analogy, just because I found an experimental, electronic, automatic nail gun, voice activated, with blinking lights, doesn't mean I should use it when building my own house, if all I need is to hammer a few nails, a regular trusted hammer will do.
That doesn't mean that you shouldn't be familiar with the basic concepts of the programming language that you use, and I feel the silicon valley bubble is sometimes unaware that not all programmers are startup hot shots.
You actually support my argument, introducing the concept of futures is just an added layer of unnecessary "cleverness" that could be avoided with quasar/pulsar.
In no way do I endorse the use of futures or whatnot, nor are they my "favourite programming paradigm".
I am simply advocating the simplest solution that works, and avoiding constructs like promises/futures seems like a good idea in that regard.
To put it differently, can I make 10000 concurrent HTTP requests (to different domains), using a non-NIO HTTP client library, without ending up with one OS thread per request?
If I can, how does the scheduler manage it?
No. We provide you with a standard HTTP client API (JAX-RS client) that gives you a blocking API. Under the hood it uses asynchronous IO. We then transform callbacks to fiber-blocking operations. So you use a standard blocking API, that is implemented asynchronously.
JDBC is a little more complicated as there is no async JDBC standard. What we do, then, is run the thread-blocking call in a separate IO workers pool. Those worker threads will block, but your API call will just block the fiber, letting other fibers use the same OS thread for something else until the JDBC call completes, at which point the IO worker will wake up your fiber.
What happens if a thread loads from a MappedByteBuffer, and the OS needs to read from disk to satisfy the load? What happens if a thread loads from some far corner of main memory that has been paged out, and the OS needs to read from disk to satisfy the load?
Those situations go beyond the power of lightweight threading systems i have seen before. They're not fatal problems, nor even serious ones for most programs, but they're part of the reason that lightweight threading hasn't, and can't, become the general solution to threading. Well, not until scheduler activations make a comeback, at least. That doesn't mean that lightweight threading is not a hugely valuable thing to have on an opt-in basis, though, as this demonstration, er, demonstrates.
Yes (to handwritten integration), but it's incredibly simple. We can transform any callback-based API to a fiber-blocking API within hours of work.
> What happens if a thread loads from a MappedByteBuffer, and the OS needs to read from disk to satisfy the load? What happens if a thread loads from some far corner of main memory that has been paged out, and the OS needs to read from disk to satisfy the load?
The thread will be blocked, but it isn't likely to be much of a problem. It's perfectly OK for the fiber to block the thread occasionally (hopefully rarely) – the work-stealing scheduler can deal with that because it runs in a thread-pool; if one thread blocks other will steal its work and do it. It's just not OK for fibers to block their thread very often. The scenarios you've described involve missed caches and so are rare by design.
> Those situations go beyond the power of lightweight threading systems i have seen before. They're not fatal problems, nor even serious ones for most programs, but they're part of the reason that lightweight threading hasn't, and can't, become the general solution to threading.
I agree. Quasar fibers are by no means meant to serve as a replacement for threads. They are specifically targeted for cases when you want lots of concurrent "threads" that interact very often by passing information (either via messages or a shared data structures), and so block and wait for each other a lot.
If you have a long-running computation: use a plain thread.
Actually, one of the cool things in Quasar is an abstraction called a strand. A strand is simply either a thread or a fiber. All of the synchronization mechanisms (channels, condition variables etc.) provided by Quasar work with strands - not directly with fibers - so you can use them both in fibers or threads. In fact, you can run a Quasar actor in a thread rather than a fiber.
Should libraries be quasar-aware to not end up in the IO workers pool when invoked?
Once the documentation is complete you'll know how to transform any callback-based asynchronous call into a fiber-blocking one, so you could integrate your own libraries with Quasar. It's very-very simple.
I guess this does still require the `await` word, but I think it's good to have a magic word for "suspend execution now" so you can see where it's happening.
An OS thread still has to block on blocking IO calls somewhere, of course. There's no "syntactic" fix for that on the JVM - you have to actually port the blocking IO to nonblocking IO - AFAIK nobody can magically fix JDBC to be nonblocking from outside JDBC.
Other than working for all JVM languages, Quasar fibers are more general in that they can spawn many functions (they have a stack), while async is limited to a single expression block. Because of this, we can hide the "await" deep inside the JDBC call stack.
Under the hood, they are similar: both instrument your code. Only async does this at the language level (it's a Scala macro), while fibers do it at the bytecode level.
That's untrue. With Akka's pipe pattern, you can take the results of any future and pipe it back to the sender, including using a map on the future if you like. This is how we do reactive database calls in Akka. For example:
At no point does this actor block. Assuming you even have something like Play calling this actor, you wouldn't be blocking there, either, you'd take the result from the actor, likely map it to a result, and Play would asynchronously return that. My basic rule is that if you're typing Await.result anywhere in your Play/Akka code, you're doing it wrong.