News for hackers. Stuff that matters.
Re:Einsum Is All You Need – Einstein Summation in Deep Learn... by yorwba on Saturday February 02, 2019 @07:06PM (#19066777)
The problem with einsum is that you have to explicitly specify the mapping between dimensions and indices every time, without any way to enforce consistency. It would be more ergonomic if each tensor had labeled dimensions. That would prevent the kind of silly mistake where you mix up the ordering of dimensions and only notice it when you later change the shape of the tensors so the different dimensions no longer match up.
Re:Einsum Is All You Need – Einstein Summation in Deep Learn... by sischoel on Saturday February 02, 2019 @07:32PM (#19066900)
I think, this is something that the community is now slowly realizing. I've must have read about that multiple times recently. Some packages that seem to provide tensors with labeled dimensions.(I haven't personally worked with them):

xarray (Python): https://github.com/pydata/xarray NamedArrays.jl (Julia): https://github.com/davidavdav/NamedArrays.jl

Re:Einsum Is All You Need – Einstein Summation in Deep Learn... by claudius on Sunday February 03, 2019 @04:07AM (#19068532)
ITensor [1] has somewhat pioneered this concept in tensor networks for condensed matter physics. If I understand correctly, Uni10 [2], a similar project, is even working on a graphical interface for such networks so that you can "draw" the network and have the computer figure out the best contraction order.

In my own code I’ve recently also implemented such named indices (only had "einstein summation"-like contraction specifiers before) and they make tensor contractions so much simpler to write, especially since you can simply overload operator * and have it figure out which legs need to go together.

As a side effect, it also enforces your algorithms to make sense because you can't simply add two tensors living on different (but equal-dimensional) vector spaces together anymore.

[1] https://itensor.org

[2] https://uni10.gitlab.io/

Re:Einsum Is All You Need – Einstein Summation in Deep Learn... by mcabbott on Sunday February 03, 2019 @05:29AM (#19068817)
I made a thing to audit that you are consistent with your indices [1] as an alternative to explicit named-tensor objects. And found, but have not used, another package in a similar spirit [2].

Although to be honest, simply writing A[n,μ,ν,c] etc. (using different letters for different spaces) makes it pretty easy to visually check that you are getting this right. This is one of the attractions of the notation, even on paper. Unfortunately np.einsum's string notation makes this harder to see, as the indices aren't adjacent to the variable name.

[1] https://github.com/mcabbott/TensorCast.jl#checking (Julia)

[2] https://github.com/ofnote/tsalib (Python)

Re:Einsum Is All You Need – Einstein Summation in Deep Learn... by claudius on Sunday February 03, 2019 @06:34AM (#19069050)
> I made a thing to audit that you are consistent with your indices [1] as an alternative to explicit named-tensor objects. And found, but have not used, another package in a similar spirit [2].

Cool, I will have to check this out!

> Although to be honest, simply writing A[n,μ,ν,c] etc. (using different letters for different spaces) makes it pretty easy to visually check that you are getting this right. This is one of the attractions of the notation, even on paper. Unfortunately np.einsum's string notation makes this harder to see, as the indices aren't adjacent to the variable name.

The problem is not so much the letter-space association but also handling the ordering of the spaces inside the tensor. For example in my code, to do a contraction over two indices, you could do prod<2>(a, b, "tlx,tr,p1,p2|tlx,p1,tl|tl,p2,tr") where the result would then have index order tl,p2,tr. The problem was then that changing the index order in one place (e.g. for performance reasons) meant having to re-check all other places where this is used. If you want to contract a tensor network like (d) in [1], this quickly gets complicated. With named indices, the above becomes a * b and if I change the index order in any place, it gets automatically changed there, too.

[1] https://journals.aps.org/prb/article/10.1103/PhysRevB.81.165...

Re:Einsum Is All You Need – Einstein Summation in Deep Learn... by mcabbott on Sunday February 03, 2019 @07:05AM (#19069163)
Right, changing the order would be a pain. Although if the reason for doing so is memory layout (for speed), then a lazy permutedims(A) would decouple index order from this. Perhaps when creating a tensor for the first time there ought to be a way to specify the layout? Haven't thought much but something like A[n^4, μ,ν, c^1] := .... would not be hard to do.

In your prod<2>(a, b, ... example, if p1 and p2 are in the same space, how would a*b know which one to contract? Or do they have different names from when a was created?

Re:Einsum Is All You Need – Einstein Summation in Deep Learn... by claudius on Sunday February 03, 2019 @08:08AM (#19069376)
> In your prod<2>(a, b, ... example, if p1 and p2 are in the same space, how would a*b know which one to contract? Or do they have different names from when a was created?

ITensor introduced this concept and I mostly just followed their lead – spaces have unique names and tensor legs have a name label and a "prime level". So for example an operator O: A → A would have one leg labelled a[uuid]-prime0 and another leg a[uuid]-prime1. Similar to how one might write O: A → A has elements O_{a a’} when writing it down on paper.

Re:Einsum Is All You Need – Einstein Summation in Deep Learn... by adamnemecek on Saturday February 02, 2019 @08:31PM (#19067180)
Re:Einsum Is All You Need – Einstein Summation in Deep Learn... by curiousmiro on Sunday February 03, 2019 @10:30AM (#19070063)
See HarvardNLP's namedtensor which is under active development!

disclaimer: i'm a contributor

Re:Einsum Is All You Need – Einstein Summation in Deep Learn... by CuriousSkeptic on Sunday February 03, 2019 @05:39AM (#19068857)
Why is it that mathematicians are so comfortable with using ordering to encode semantics?

Evident from this thread it seems many developer recognize the need to use proper labels to simplify reasoning. Doesn’t mathematicians have similar desires?

Re:Einsum Is All You Need – Einstein Summation in Deep Learn... by improbable22 on Sunday February 03, 2019 @06:12AM (#19068975)
Well, many different notations are convenient in different contexts. The compiler they target is a bit more flexible than developers', so they are free to just describe what they are doing on page 1.

Human languages face the same trade-off: in some word order is very important, in others less so, but they need to compensate with some kind of case labels.