News for hackers. Stuff that matters.
Re:Matrix multiplication inches closer to mythic goal by melling on Tuesday March 23, 2021 @11:22AM (#26556080)
I am doing Andrew Ng’s ML Coursera course with Matlab.

I’ve now got the crazy desire to see matrices and vectors built into every language in a clean and succinct way.

Re:Matrix multiplication inches closer to mythic goal by tasty_freeze on Tuesday March 23, 2021 @11:45AM (#26556381)
One of the unexpected things is Dartmouth BASIC from 1964 had matrix primitives, despite being an otherwise primitive language. MS never took up the feature, but I know Wang BASIC did on their 2200 family of machines.

For example:

    10 DIM A(3,3),B(3,3),C(3,3)
    20 MAT READ A
    30 DATA 1,-5,7, .45,23,9, -5,-1,0
    40 MAT B=A:REM assignment
    50 MAT B=ZER:REM zero array
    60 MAT B=CON:REM all 1s
    70 MAT B=IDN:REM identity
    80 MAT B=TRN(A):REM transpose
    90 MAT B=(2)*A:REM scale
    100 MAT B=INV(A),D:REM invert array, determinant in scalar D
    110 MAT C=A*B:REM matrix multiplication
    120 MAT PRINT C:REM prints something close to an IDN array
Re:Matrix multiplication inches closer to mythic goal by Someone on Tuesday March 23, 2021 @01:41PM (#26557822)
The first version of Dartmouth basic ran on a system with a whopping 20kB of memory (8k words at 20 bits each. See https://en.)wikipedia.org/wiki/GE-200_series)

The GE-635 (https://en.wikipedia.org/wiki/GE-600_series) that it ran on most of its life could have 4 CPUs, had protected memory and floating point hardware, and supported at least 200 kilowords of memory (http://www.bitsavers.org/pdf/ge/GE-6xx/CPB-371A_GE-635_Syste...)

At 36 bits/word, that’s 900 kilobytes.

The first version of Microsoft basic (https://en.wikipedia.org/wiki/Altair_BASIC) ran in 4 kilobytes (leaving about ¾ kilobytes for programs and data), so it’s understandable that it lacked some features (it didn’t even have else clauses in if statements, had SIN, but not COS or TAN, didn’t support integer or string variables, etc))

Re:Matrix multiplication inches closer to mythic goal by tasty_freeze on Tuesday March 23, 2021 @09:47PM (#26562609)
No question it would not make sense in that original 4K or even 8K BASIC, but MS produced BASICs with much larger memory budget and the matrix ops never appeared (as far as I know).
Re:Matrix multiplication inches closer to mythic goal by Someone on Wednesday March 24, 2021 @03:54AM (#26564574)
They added tons of stuff that made business sense to them. Apparently, matrices as built-in types didn’t. Nowadays, the language is powerful enough to implement them as a library (e.g. https://www.freevbcode.com/ShowCode.asp?ID=3682. Operator overloading became possible at some time, too. https://docs.microsoft.com/en-us/dotnet/visual-basic/program...)

It’s not as if they don’t know matrices exist. Excel has matrix operations (https://support.microsoft.com/en-us/office/mmult-function-40...)

Re:Matrix multiplication inches closer to mythic goal by coliveira on Tuesday March 23, 2021 @12:02PM (#26556605)
Believe it or not, engineers were heavy users of BASIC in the first few years. And FORTRAN was created with the notion of matrix multiplication, so it was just natural that BASIC implementations supported matrix multiplication.
Re:Matrix multiplication inches closer to mythic goal by pavlov on Tuesday March 23, 2021 @02:32PM (#26558484)
My grandfather was a construction engineer. I remember visiting his office around 1990 just before he retired.

His only computer was a Commodore 64 and he had written all the software himself in BASIC over the years, just translating a lifetime of professional experience into these little interactive programs on floppies.

Maybe the modern equivalent would be to use Excel.

Re:Matrix multiplication inches closer to mythic goal by mamcx on Tuesday March 23, 2021 @04:50PM (#26560239)
Yeah, this is something pretty useful. I also add to even go beyond and go full relational, that is what I'm trying with https://tablam.org
Re:Matrix multiplication inches closer to mythic goal by montalbano on Tuesday March 23, 2021 @12:25PM (#26556860)
Convenient matrix algebra is one of the selling points of Julia.
Re:Matrix multiplication inches closer to mythic goal by nerdponx on Tuesday March 23, 2021 @12:45PM (#26557094)
I'm really hopeful that Julia will continue to gain adoption.

That said, I would love to have good data science library support (bindings to stuff like BLAS/LAPACK, Torch, Stan, etc) in Idris.

Imagine vector and matrix dimensions checked at compile time. Probabilities guaranteed to be between 0 and 1, checked at compile time. Compile-time enforcement that you are handling "nil/null" data inputs correctly when reading data from a database or CSV file. Writing a web scraper with linear types to help catch bugs & performance leaks. Maybe even proving theorems about the math code you're writing ("is this a valid way to express X"?).

Maybe not ideal for rapid exploratory model development, but it'd be pretty damn cool for writing data science tooling/libraries and "production" ML code in my opinion.

Re:Matrix multiplication inches closer to mythic goal by kristjansson on Tuesday March 23, 2021 @10:18PM (#26562812)
Not that it has bindings to other tools, but it sounds like Dex[1] would be relevant to your interests!

[1]: https://github.com/google-research/dex-lang

Re:Matrix multiplication inches closer to mythic goal by Ericson2314 on Tuesday March 23, 2021 @07:26PM (#26561667)
Looking forward to Dependent Haskell for linear algebra, and seeing what existing stuff can be retrofit.
Re:Matrix multiplication inches closer to mythic goal by taxemeEvasion on Tuesday March 23, 2021 @12:03PM (#26556607)
It's a main reason why Fortran still has such a hold on the SciComp/HPC community.
Re:Matrix multiplication inches closer to mythic goal by gowld on Tuesday March 23, 2021 @12:29PM (#26556895)
Fortran is also optimized and debugged over decades of testing.
Re:Matrix multiplication inches closer to mythic goal by karma_fountain on Tuesday March 23, 2021 @12:33PM (#26556942)
Yeah those algorithms are numerically tested, haven't changed in years, and really fast. You can certainly get to convenient matrix manipulation in C++, but then your compile times will increase significantly.
Re:Matrix multiplication inches closer to mythic goal by DesiLurker on Tuesday March 23, 2021 @01:45PM (#26557860)
Can the fortran implementations do auto-vectorization on simd hardware? or map to gpu using cuda-like library?

not rhetorical, genuine question.

Re:Matrix multiplication inches closer to mythic goal by cygx on Tuesday March 23, 2021 @01:53PM (#26557965)
Re:Matrix multiplication inches closer to mythic goal by TheRealKing on Wednesday March 24, 2021 @12:18PM (#26569227)
Fortran is a fully-vectorized and parallelized language at all levels, from instructions to shared and distributed parallelism (corresponding to one-sided MPI communications). This has been the case for at least 3 decades if not more. There are mature GPU support for Fortran. NVIDIA and PGI are currently implementing the standard parallel features of Fortran via GPUs.
Re:Matrix multiplication inches closer to mythic goal by jhayward on Tuesday March 23, 2021 @05:33PM (#26560674)
Yes, in fact all the original research for those things was done in Fortran compilers.
Re:Matrix multiplication inches closer to mythic goal by Zardoz84 on Tuesday March 23, 2021 @01:34PM (#26557727)
You can have a fast implementation and fast compilation time with DLang
Re:Matrix multiplication inches closer to mythic goal by andi999 on Tuesday March 23, 2021 @01:37PM (#26557757)
Performance might be worse in c++
Re:Matrix multiplication inches closer to mythic goal by teruakohatu on Tuesday March 23, 2021 @03:43PM (#26559459)
The base type in R is a vector. A number such as 3.14 is simply a vector of length 1. So functions are vectorized by default (as long as you don't do any operations that are explicitly not vectorizable).

Once you get your head around everything being a vector, R can be a lot of fun.

I love Julia but I wish I didnt need to explicitly broadcast functions across a vector.

Re:Matrix multiplication inches closer to mythic goal by cbkeller on Tuesday March 23, 2021 @07:11PM (#26561547)
I was annoyed by that at first, but eventually came to appreciate how Julia's semantics for when broadcasting is or is not required for operations on scalars, vectors, and matrices map to the underlying math. I.e., for an N x N matrix `A`

  5 * A
does not require broadcasting because multiplication of a matrix by a scalar is mathematically well-defined, whereas

  5 + A
(assuming you want element-wise addition) does require broadcasting, i.e.,

  5 .+ A
because addition of a vector and a matrix is not mathematically well-defined.
Re:Matrix multiplication inches closer to mythic goal by teruakohatu on Tuesday March 23, 2021 @09:06PM (#26562337)
I do understand why they do it, and numpy can play a bit fast and loose which can be confusing, but I miss reading elegent vector-by-default R code.

The Julia approach also has some drawbacks, I was writing some code the other day and forgot to broadcast exp() across a matrix. exp(A) and exp.(A) gives very different results but it's not immediately obvious looking at code, especially when the code is a mixture of broadcasted and native matrix function calls which dots peppered all over the place.

Re:Matrix multiplication inches closer to mythic goal by eigenspace on Tuesday March 23, 2021 @10:10PM (#26562761)
Well, again I'd say that's just like the above examples. The exponential of a (square) matrix is indeed a very well defined object [1] with semantics perfectly consistent with the exponential of a scalar. On the other hand, broadcasting exp over a matrix is a very different operation, which doesn't really have any meaning in a linear algebra sense.

I understand it's an easy mistake to make when you're used to R or Numpy, but I'd say it's their fault you're making this mistake, not Julia's.

[1] https://en.wikipedia.org/wiki/Matrix_exponential

Re:Matrix multiplication inches closer to mythic goal by teruakohatu on Tuesday March 23, 2021 @11:35PM (#26563296)
If you re-read my comment you will see I am not criticizing the language, it makes mathematical sense.

I was pointing out one downside of the broadcast syntax, is that a typo can lead to non-obvious errors in calculations. R avoids this with a expm() function.

Don't get me wrong, I love using Julia, it makes turning maths into code easy, and this is not even close to a deal breaker.

Re:Matrix multiplication inches closer to mythic goal by eigenspace on Wednesday March 24, 2021 @01:47AM (#26564003)
Sure, but my point is that this cuts both ways.

E.g. I could just as easily write

"one downside of autovectorization, is that a typo can lead to non-obvious errors in calculations. When I want the exponential of a matrix, I might accidentally write exp(M) and get unwanted vectorization. This is a typo that can lead to non-obvious errors in calculations. Julia avoids this with a exp.(M) syntax."

I think the potential for typos is symmetric here between julia and R. I think what's asymmetric though is that Julia's broadcasting is more extensible, less and hoc and more general than R's auto-vectorization since it can be done over arbitrary containers and you can control precisely what calls broadcast.

______________________________

Sorry if this comes off as confrontational, that's not my intention. I just think this is a very interesting and fruitful corner of language design.

Re:Matrix multiplication inches closer to mythic goal by make3 on Tuesday March 23, 2021 @02:30PM (#26558453)
Yes, have you tried Julia? It is pretty much how you describe. There are other things that I don't like about that language, but the matrix part is really great
Re:Matrix multiplication inches closer to mythic goal by mkaic on Tuesday March 23, 2021 @03:11PM (#26559043)
Just finished the course and couldn't agree more. Everything seemingly is faster with matrices!
Re:Matrix multiplication inches closer to mythic goal by centimeter on Tuesday March 23, 2021 @01:26PM (#26557596)
Beyond matrices and vectors, it would be nice to have generalized (typed) tensors. What you can do with matrices alone is really somewhat limited.
Re:Matrix multiplication inches closer to mythic goal by Robotbeat on Tuesday March 23, 2021 @02:47PM (#26558696)
From a data structure perspective: You mean like an n-dimensional array? That’s what Matlab does. Matrices (2D arrays) are just a special case of array in Matlab.
Re:Matrix multiplication inches closer to mythic goal by whimsicalism on Tuesday March 23, 2021 @03:42PM (#26559445)
I suspect by the usage of the phrase "typed" they mean something like that, but where the axes/dimensions/whatever you call them are typed - ie. you can't naively take a product along the time dimension of A with the batch dimension of B.
Re:Matrix multiplication inches closer to mythic goal by jpeloquin on Tuesday March 23, 2021 @03:57PM (#26559637)
xarray seems to fit the bill. As long as you're ok with runtime axis/dimension/type checking.

https://xarray.pydata.org/en/stable/

Re:Matrix multiplication inches closer to mythic goal by whimsicalism on Tuesday March 23, 2021 @08:21PM (#26562050)
I've explored this space a bit, and I don't think the perfect solution is here yet. In my view, static checking should be the goal. It's harder to iterate quickly when your network dynamically fails 30 layers later due to some edge case where you forgot to take a transpose. Definitely much better than it silently succeeding if the dimensions happen to be the correct size though!

https://github.com/deepmind/tensor_annotations seems promising. I've developed a mypy plugin for pytorch that does similarly off of the "Named Tensor" dynamic feature (which isn't well supported yet), but haven't released it yet.

I'm also excited by the ways in which including this in some sort of "first-class" way could make tensor semantics simpler, which I'm assuming xarray allows? Many of the operations enabled by http://nlp.seas.harvard.edu/NamedTensor are quite nice and similar ideas can let you write more generic code (ie. code that works automatically for on tensors with and without a batch dimension)

Re:Matrix multiplication inches closer to mythic goal by bobbybabylon on Tuesday March 23, 2021 @02:22PM (#26558349)
Have you looked at languages like J or APL?