This is also very easy in Keras / Tensorflow using the FloatX parameter, or specifying e.g. float16 dtypes.

However, I’d say desiring a framework that allows “easy” extensibility to choose float precisions lower than 16 bits and have it “just work” is actually a mistake. That type of flexibility is overkill.

Instead, supporting a limited set of fixed types is better. To experiment with a new type requires some integration hurdle to make it recognized by the backend, and then requires published research or some similar type of evidence that there are use cases which materially benefit from that new additional fixed data type, to get a PR approved to add it.

The reason is that permitting arbitrary complexity growth in the form of “easy” custom data type support has two big downsides, (a) the mechanism that makes it easy had to consume maintenance and development resources even if it’s a very obscure form of customization, and (b) more importantly, it proliferates and worsens the already insane problems of being able to export / import models from one language/framework to another.

It’s a case study of KISS and YAGNI: this is super premature abstraction especially if it’s for experiments. And the hurdle of making a branch and adding your new dtype in the backend is not (and should not be seen as) a significant engineering hurdle. Rather it’s a very good check on complexity growth.

Yeah except flux code is way simpler than tensorflow code, both for the end user and internally as well. It's not a premature optimization, it comes "for free" in Julia. Besides, you don't know what someone might need. Say someone wants to implement a deep learning model with complex valued activations or quaternion valued activations. What then?

There is no complexity added in flux to support arbitrary datatypes; Character level lstm in about 30 lines of flux. The flux library itself is a very, very small library. Converting character lstm to a custom datatype is about three lines of code (plus about 60, reusable, for the datatype).

This speaks to the good choice of abstractions in Julia. What you may call unnecessary optimization is for me critical research since I'm investigating building hardware and I want to make sure the fp type (and it's not at all a standard IEEE type) I would implement is usable. Most deep learning is memory bandwidth limited so a decrease in bitsize has an O(n^2) effect in computation speed.

In the spirit of rapid iteration it was far preferable to implement 50 lines of code in Julia to get my type able to do machine learning and then know if it failed or succeeded rather than write a tf kernel, which probably would have taken me months.

> “Say someone wants to implement a deep learning model with complex valued activations or quaternion valued activations. What then?”

This sounds like premature abstraction to me...

There is work on rotationally invariant networks, e.g. for identifying galaxies, or cells under a microscope. For example:

https://arxiv.org/abs/1612.04642

https://arxiv.org/abs/1805.12301

I haven't looked closely enough to be sure if they literally had complex activations, but this seems like an obvious use. Maybe they would have, if only tensorflow made it easy.

Why would you ever want to represent activations directly as complex numbers in that case? I can’t think of any good reason, compared with putting them in rotation matrix form or some other equivalent form that actually maps to the domain modeling problem.

Even when working in signal processing problems that require complex arithmetic, the underlying representations are just based on tuples of doubles and operator conventions, and you always need to map to real spaces (real part, imaginary part, angle, or magnitude) for any type of analytical representation that can be human readable.

In all these cases, the idea that what we should optimize for is overhead-free easy expression of cutesy math domain verbiage is a bad idea.

Writing libraries that expose an API that matches the user’s domain mental model is a great thing. But enforcing a particular abstraction and extensibility hierarchy so those things can be “autogenerated” just by parameterizing over a new type turns out to be actually much worse than just writing that type separately, with helper functions and converters, and customizing its API to be efficient from a domain mental model perspective.

A better way, for example, might be to use mixin patterns or decorators and other metaprogramming, while writing a custom data type and its associated methods.

Something like this? https://github.com/andyferris/Traitor.jl

That is definitely a cool direction to take it!

How is it premature abstraction if it takes zero extra lines of code to support it and have it optimized? That's kind of the beauty of Julia.

It didn’t require zero lines of code. It required a huge amount of backend code to set up the abstraction and make lots of built-in types that adhere to the abstraction. And in cases when the abstraction fails to offer the exact type of extensibility needed (which is most of the time unless you’re authoring yet another highly abstracted library that can tie its use cases to that underlying abstraction, which is never in practice), then it was wasted effort, and “no overhead” is a false description, because you still have to dig into the guts of all the stuff that gets auto-generated if you plugged into the abstraction and change the mechanism of how it gets auto-generated for your special case, or else (usually easier), just write separate data structures outside of the abstraction vortex and have a few small converters or helpers that marshal your custom data type into and out of the abstraction for the really tiny anount of auto-generated features that actually matter to the use case.

The “but it requires zero lines of code” thing is so misleading once you hit real use cases where the choices of how the abstraction auto-generates things end up being unusable for some specific situation.

Right here we have a classic case of someone on the internet anonymously saying something is impossible to do while there are many many examples of exactly this working. I recommend readers of these posts to ignore the FUD and do some Google searches to look through some Julia code repositories to see it in action. There are some great tutorials and fact-based discussions out there that can lead you to some useful examples with tricks you can employ in your own code.

It looks like you are just posting knee-jerk defensive posts about julia I guess. Whatever this is, it’s clearly not related to my earlier comments in the thread.

What are you talking about? Where did I say any of this was not possible? It’s obviously possible.

It just turns out to be bad when you do it. It causes problems that the company line memo about zero overhead never is upfront about.

Premature abstraction is a problem for engineering.

But what about scientists who are not too fussed with engineering considerations but would like to explore such things? Then this extensibility can be valuable.

At some point one of those explorations becomes useful and it's no longer premature.

I can see where you're coming from in any other language. But in Julia it's important to realise that this "premature abstraction" isn't actually any extra work, it's just the default. If we write `f(x) = x+x` then `f` takes anything that can be added, which can be any custom number or matrix type, or really anything else. Adding type restrictions to make it work with only a limited set of types is completely doable, but actually more work than just leaving it generic.

We didn't at any point decide "it's worth the extra effort/complexity to make Flux work with custom number types"; it's just inadvertently been that way from day one, and I didn't even know anyone one was making use of it until today.

This is the same for many languages that treat operators with type class patterns. It’s still usually bad design choice. I find a lot of language designers & programmers like it, but they are super disconnected from the realities.

For an example consider breeze and spire in Scala. There’s so much effort to create these bloated numeric type hierarchies that abstract out things like monoids, rings, fields, iterability, sortability, etc.

It’s not good. Just having really boring repetitive implementations for each distinct data structure would be better! No joke! Being able to write type generic functions over sortable matrix subclasses turns out to not be valuable unless you’re also writing a highly abstracted library, which is never, certainly not when you’re using it for experiments.

Nobody needs to be able to make a DenseMatrix[Quaternion] and get it to automatically pick up implementations of fancy indexing. No. You can just write your own helper methods, and this is better, more convenient, applies less pressure for DenseMatrix to have some indecipherably complicated abstract implementation so it can be more free to just specialize on linear algebra functionality that works for DenseMatrix[Double] which is what is needed 99.999999999% of the time.

You really should try Julia, before making claims about its complexity.

The numeric type systems are simple, and designed for convenience, not to satisfy mathematical theory. In the case of FloatX, It's basically Any <: Number <: AbstractReal <: AbstractFloat <: FloatX

For complex datatypes, like vectors, matrices, dicts, etc, you have templatable datatypes, but that is no more complex than C++, and actually far cleaner in implementation.

For the most part, you do not NEED to make a Matrix{Quaternion}. And that's fine. However, if you do, the standard library will do the right thing, as if you had made a Matrix{Int32} or a Matrix{8BitGaloisField}. And if you choose to use Matrix{Float32}, the type system interacts with the compiler, and in the standard library it picks up the fortran BLAS library so you get faster-than-c performance.

On the other hand, you might be deploying a really large matrix on a supercomputing cluster, and it might be useful to re-index the matrix as a datatype that fits in the L1 cache of your Knights Landing chips. In which case, you have the option of redeploying as an AbstractMatrix{Float64}, implementing index catching functions, and dropping it in to you code (probably about 100 lines of code total, if even) without having to rewrite every single matrix operation everywhere.

It’s so funny to me how Julia proponents often make it an ad hominem attack as if the writer hasn’t used the language. I’ve been using and following Julia closely since late 2012, and even attended a few meetups / talks at MIT about it since I was a grad student at the time, and even took a random matrices class with Alan Edelman in which he talked quite a bit about early julia.

Julia is by no means the only language to have patterns like this either, and in fact it’s not even a language where these patterns are particularly easy to use (I would reserve that for Haskell, but admit there may be other languages I don’t know which also make the cut — not julia though).

Your two ending paragraphs read to me like a super naive restatement of the company line memo for why these types of parametric abstractions are supposed to be good. It’s like a political platform, and just like a political platform it doesn’t keep its promise.

I have worked on projects where we needed to customize bit packing, not for cache performance, but for control over a modified version of sparse matrix types.

And I’m telling you the idea that we’d ever rely on the language’s chosen abstraction and do something like AbstractSparseMatrix{Float64} to pick up a bunch of interface properties “for free” while making the underlying logic specialized for our sparse format is crazy. It’s a naive false promise that grad students believe and it gets quickly beaten out of them in the real world once you realize how the type constraints and inheritance / type class extension constraints this places on you are too limiting and end up requiring just too much boilerplate that can’t quite be autogenerated because the way the abstract interface was chosen just doesn’t quite match your use case.

Finally you realize going down this road was the wrong idea all along, and you just write a super short implementation of MyCustomSparseMatrix or MyCustomCachePropertyMatrix in your case, and you fill in the logic manually that you thought you’d be clever by getting “for free” via plugging into some abstraction hierarchy, and often realize for your use case you don’t need to re-implement hardly any of it, and can do the boring parts pretty easily with converters or helper functions that marshal between whatever “for free” functionality you hoped to get and your simple custom not-parametric-abstraction type.

I’ve been down this road too many times, in many languages. I just leave it for the grad students who like playing with abstraction toys, and instead I just get back to actual work, solving problems economically, which warrants a super strong heuristic of avoiding this type of parametric abstraction pattern as much as possible.

I'm using the abstract numbers for encodings of uncertainty and probability distributions. This goes far beyond FloatX types and is really helping spawn a whole new area of research. The number type abstractions in Flux are really one of a kind. Kudos to the developers.

I work on large scale MCMC and causal inference problems. Would be very interested to know what this area of research is where you require abstract numeric types themselves to represent uncertainty, and why it would be different than other inference algorithms that handle uncertainty. I admit, at first blush I am extremely skeptical. It sounds like a silly sort of thing where instead of parameterizing over a numeric type, you might parameterize over a number-from-a-distribution type, and then try to make the type system represent how everything will flow in an MCMC setup, similar to tools like pymc, except where those are declarative and procedural, this would attempt to embed that into the type system. I can scarcely think of a worse way to represent manipulating uncertainty though. I hope I’m just reading your comment incorrectly.

It's for differential equations. Getting uncertainty without parameter sampling saves a lot of computational time and really opens up the problems that can be solved.

But why does “getting uncertainty without parameter sampling” have anything to do with parametric numeric types?

The latter is just a possible manner of implementation (that I’d argue is too cutesy), but there are many other ways to design a system like that, for example like fused types in Cython combined with numba class jitting.

I still see no reason to believe that something that parametrizes differential equation functions or linalg functions over “uncertainty primitives” would be anything but a functional programming hot take on something that could be more straightforwardly done many other ways not relying on parametric abstraction.

The Julia solution didn't require anyone to actually think about making it work, and it works well (we found out from a Discourse post that it works, the developers didn't even know :)), and now we are using it in our research codes because it is a great way to speedup what was traditionally done via parameter sampling.

Until I see someone else take an existing ODE solver like LSODA and convert it into something that can output uncertainties without having to do parameter sampling, I won't think other ecosystems are very close to what we have already done. Places like SciPy are still calling out to Fortran routines from ODEPACK for this, so making it work with Numba class jitting is a long way away. Show how easy it is to code it by showing code. Ours is already done: the ball is in your court.

I just looked up your paper with Nie and indeed you’re just using a particular set of patterns with multiple dispatch and metaprogramming. Nothing is fundamentally different than other ways of implementing the same thing that don’t rely on parametric abstraction.

You seem not to know about numba and Cython given that you responded with a comment about scipy using FORTRAN, which is not relevant. You can do the exact same multiple dispatch patterns with Cython fused types, and with several dispatch techniques in numba.

Look, I’m glad people like your library. It doesn’t change the larger points about this type of design pattern being premature abstraction.