This sounds like premature abstraction to me...
https://arxiv.org/abs/1612.04642
https://arxiv.org/abs/1805.12301
I haven't looked closely enough to be sure if they literally had complex activations, but this seems like an obvious use. Maybe they would have, if only tensorflow made it easy.
Even when working in signal processing problems that require complex arithmetic, the underlying representations are just based on tuples of doubles and operator conventions, and you always need to map to real spaces (real part, imaginary part, angle, or magnitude) for any type of analytical representation that can be human readable.
In all these cases, the idea that what we should optimize for is overhead-free easy expression of cutesy math domain verbiage is a bad idea.
Writing libraries that expose an API that matches the user’s domain mental model is a great thing. But enforcing a particular abstraction and extensibility hierarchy so those things can be “autogenerated” just by parameterizing over a new type turns out to be actually much worse than just writing that type separately, with helper functions and converters, and customizing its API to be efficient from a domain mental model perspective.
A better way, for example, might be to use mixin patterns or decorators and other metaprogramming, while writing a custom data type and its associated methods.
The “but it requires zero lines of code” thing is so misleading once you hit real use cases where the choices of how the abstraction auto-generates things end up being unusable for some specific situation.
What are you talking about? Where did I say any of this was not possible? It’s obviously possible.
It just turns out to be bad when you do it. It causes problems that the company line memo about zero overhead never is upfront about.
But what about scientists who are not too fussed with engineering considerations but would like to explore such things? Then this extensibility can be valuable.
There is no complexity added in flux to support arbitrary datatypes; Character level lstm in about 30 lines of flux. The flux library itself is a very, very small library. Converting character lstm to a custom datatype is about three lines of code (plus about 60, reusable, for the datatype).
This speaks to the good choice of abstractions in Julia. What you may call unnecessary optimization is for me critical research since I'm investigating building hardware and I want to make sure the fp type (and it's not at all a standard IEEE type) I would implement is usable. Most deep learning is memory bandwidth limited so a decrease in bitsize has an O(n^2) effect in computation speed.
In the spirit of rapid iteration it was far preferable to implement 50 lines of code in Julia to get my type able to do machine learning and then know if it failed or succeeded rather than write a tf kernel, which probably would have taken me months.