To add a bit, kudos to root-parent boil-down. Programmers have already the good representation, and call it n-dimensional array, that being a list of lists of lists ... (repeat n times) ... of lists of numbers. The only nuisance is that what programmers call dimension, math people call it rank. It is the sizes of those nested lists what math people call dimensions. It's all set up for a comedy of errors. Also in math the rank is split to make explicit how much of the rank-many arguments are vectors and how many are dual vectors. You'd say something like this is a rank 7 tensor, 3 times covariant (3 vector arguments) and 4 times contravariant (4 dual vector arguments) summing 7 total arguments. I'm assuming a fixed base, so root-parent map determines a number array.

Slight disagree here -- matrices are enough for transformations in 2, 3, 4, and 100 dimensions. Tensors are not arrays with more rows and columns; they are higher dimensional objects -- more indices, not greater range of indices.

Is a tensor higher dimension, or is it the generalized form of the structure encompassing all of it (individual numbers (0 dimensions), vectors (1 dimension), matrixes (2 dimensions), and so on)? Kind of like how an n-sphere describes circles and spheres for n equal to 2 or 3.

Any given tensor has a type (p,q). Say d=p+q. Then d is the 'dimension' of the tensor in the sense that you would need nxnx...n=n^d numbers (think of an nxnx...n array) to describe the tensor where n is the dimension of the underlying vector space.

> or is it the generalized form of the structure encompassing all of it...Kind of like how an n-sphere

If you are asking whether there are examples of tensors parametrized by an integer d, you can cook up examples - like the (d,0) tensor whose input is a sequence of d vectors and just adds up all the components with respect to some basis in each slot and then adds this number across all the d slots.

But just like a n-spere is a special example, of a polynomial in n variables, the above tensor is a specific example - it is the tensor where all the components are 1 in the higher dimensional array.

Sometimes, one considers the algebra of tensors across all dimensions like the symmetric algebra or exterior algebra simultaneously (where there is multiplication operation between tensors of different dimensions), but that might not be what you were asking about.

> (individual numbers (0 dimensions), vectors (1 dimension), matrixes (2 dimensions), and so on)

You're using the word "dimension" in two (distinct) ways. Instead, use the word "rank":

> (individual numbers (0 rank), vectors (1 rank), matrixes (2 rank), and so on)

Now, we can talk about a 4-dimensional rank-1 tensor, e.g., a 4-element vector.

Now, think about a 4x4 matrix: if we multiply the matrix by a 4-vector, we get a 4-vector out: in some ways, the multiplication has "eaten" one of the ranks of the matrix; but, the dimension of the resulting object is the same. If we had a 3x2 matrix, and we multiplied it by a 3-vector, then both the rank has changed (from 2 to 1) and the dimension has changed (from 3 to 2).

A tensor has any number of rank.

More importantly, the ranks of a tensor come in two "flavors": a vector and a one-form. The concepts are pretty darn general, but one way to get a feel for how they're related is that the transpose of a vector can be its dual. This gets into things like pre- and post- multiplication; or, whether we 'covary' or 'contravary' with respect to the tensor.

Frankly, tensor products are a beast to deal with, mechanically, so the literature mostly deals with them as opaque objects. The modern tensor software libraries and high performance computing has seen a sea-change in the use of GR.

The problem is that I'm using dimensional in two different senses. A vector with N elements, for example, is a one-dimensional array of numbers operating on N dimensional spaces, and so on for matrices and tensors.

If you're doing graphics programming you're operating in three and four dimensional spaces mostly (four dimensional being projective spaces), because that's the space you're trying to render in two dimensions. But you'll rarely need anything higher than a matrix (a two-dimensional data structure) for operations on that space.

If you're doing physics you're operating in three, four, and infinite dimensional spaces, mostly. And you'll routinely use higher data structures -- even things like moment of inertia for rigid bodies can't really be described without rank 3 tensors (a three-dimensional data structure).

In statistics and machine learning, you're operating in very high dimensional spaces, and will find yourself using non-square tensors especially (in the other areas everything will be square). The data structures will generally be high dimensional as well, but usually just a function of model complexity; so maybe 4 or 5 dimensional data structures.

Is there a difference between a 4x4 matrix and a 4x4 tensor?

A matrix is a subset of a tensor and the 4x4 matrix is absolutely a tensor. Tensors can be way more complex and do more but the 4x4 matrix you use in 3D operations is a great starting point.

A matrix is an array of numbers. It usually represents a linear map (in some basis). A tensor is more like that linear map.

In short, tensors generalize matrices. While we can probably guess accurately what "4x4 matrix" means, "4x4 tensor" is missing some information to really nail down what it means.

Interestingly, that extra information helps us to differentiate between the same matrix being used in different "roles". For instance, if you have a 4x4 matrix A, you might think of it like a linear transformation. Given x: V = R^4 and y = Ax, then y is another vector in V. Alternatively, you might think of it like a quadratic form. Given two vectors x, y: V, the value xAy is a real number.

In linear algebra, we like to represent both of those operations as a matrix. On the other hand, those are different tensors. The first would be a rank-(1,1) tensor, the second a rank-(2, 0) tensor.

Ultimately, we might write down both of those tensors with the same 4x4 array of 16 numbers that we use to represent 4x4 matrices, but in the sort of math where all these subtle differences start to really matter there are additional rules constraining how rank-(1, 1) tensors are distinct from rank-(2, 0) tensors.

"4x4 tensor" is not a useful notion. A (4, 4)-rank tensor would make more sense...

Is there a similar "gimbal lock" problem that people ran into?

this is still not it. 4d would simply be a 4x4 matrix instead of 3x3

tensors are something which no one has been able to fully or adequately describe. I think you simply have to treat them as a set of operations and not try to map or force them unto existing concepts like linear algebra or matrices. they are similar but otherwise something completely different.