News for hackers. Stuff that matters.
Re:Tensor Considered Harmful Part 2 by nabla9 on Monday January 14, 2019 @07:16AM (#18902452)
Mathematically tensors are multilinear maps, not data structures. Typically tensors in deep learning represent multilinear mappings forwards and backwards. Scalars, vectors and matrices can also be tensors when they represent mapping.

If you think how normal fully connected feedforward network learns, its outer products of derivatives. Inference is affine transformations + nonlinearity sequence.

Re:Tensor Considered Harmful Part 2 by thanatropism on Monday January 14, 2019 @09:22AM (#18903102)
We've got a winner. Tensors "do" something; they're functions.

In the rank-1 and rank-2 case these functions can be represented as objects familiar to linear algebra. 3-tensors are not the same as stacked two-tensors, however.

I wonder if "neural networks as ODEs" will catch on. I could use some of my masters work and go get a PhD.

Re:Tensor Considered Harmful Part 2 by joshuamorton on Monday January 14, 2019 @12:47PM (#18904766)
>We've got a winner. Tensors "do" something; they're functions.

And a neural network is just a stack of linear transforms (the "tensors") and nonlinear transforms (the activations). So in a lot of cases, these "tensors" do do something. Granted the abstraction for storing the active tensors is also a great abstraction for storing high-dimensional data (like a 128x128x4x60 2-second video).

>neural networks as ODEs

You're a few months late :P [0]

[0]: https://arxiv.org/abs/1806.07366

Re:Tensor Considered Harmful Part 2 by twtw on Monday January 14, 2019 @10:47AM (#18903824)
Is it accurate to say that rank-n tensor is to n-D array of numbers as linear transformation is to matrix?
Re:Tensor Considered Harmful Part 2 by krastanov on Monday January 14, 2019 @11:16AM (#18904061)
For a mathematician or a physicist, yes. But the word ended up having new, confusing meanings.
Re:Tensor Considered Harmful Part 2 by nabla9 on Tuesday January 15, 2019 @08:16AM (#18910892)
It's the same for ML perspective as well.
Re:Tensor Considered Harmful Part 2 by Koshkin on Monday January 14, 2019 @07:14PM (#18907837)
> they're functions

Well, in math a lot of things we normally do not think of (or use) as a function can still be considered as such. For example, a vector "is a function." Why? Because it does something (to a covector, or a linear form - which, in turn, is a function, etc. etc.) Even a number "is a function" by a similar reasoning. Objects in math are very flexible conceptually.

Re:Tensor Considered Harmful Part 2 by thanatropism on Monday January 14, 2019 @10:09PM (#18908589)
Ehhh.

Take for simplicity a constant real number. A constant is not a function but there's a function from the reals to a space of real valued functions. Likewise every vector can be converted into a cove tot, but...

You can say these function-valued functions are bijective so you can work on either side of the fence. Your can also work with equivalence classes if you don't quite have a bijection. But a lot of mathematics is about being specific about what we want, even if we're lazy in writing later and smush related concepts together in the same symbol.

Re:Tensor Considered Harmful Part 2 by amelius on Tuesday January 15, 2019 @05:17AM (#18910090)
> Mathematically tensors are multilinear maps, not data structures.

Yes, but any parameterized operator needs data for its representation, similar to how a closure is code + data.