Llama2.java: Karpathy's llama2.c ported to Java

Re:Llama2.java: Karpathy's llama2.c ported to Java by gavinray on Tuesday August 08, 2023 @11:02AM (#37050024)

The Java code is impressively written, using newer features like MemorySegment.

Looked at the author and realized it's Alfonso from the Graal team -- makes sense.

I wonder whether the "matmul" code could be further optimized with the Vector API and SIMD.

[ Parent | on HN ]

Re:Llama2.java: Karpathy's llama2.c ported to Java by kurhan on Tuesday August 08, 2023 @12:42PM (#37051787)

Also new virtual threads might be beneficial. I was experimenting using Vector api for matrix multiplication once and effect was pretty good.

[ Parent | on HN ]

Re:Llama2.java: Karpathy's llama2.c ported to Java by mike_hearn on Tuesday August 08, 2023 @03:08PM (#37053990)

Virtual threads shouldn't help as the program isn't I/O or wait bottlenecked. It's a pure computation, so it's all about vectorization here.

[ Parent | on HN ]

Re:Llama2.java: Karpathy's llama2.c ported to Java by mukel on Tuesday August 08, 2023 @03:22PM (#37054187)

Author here: I implemented several versions of matmul with different unrolling schemes using the Vector API and I got a ~4X speedup with a single thread, but the speedup fades the more threads you add. I think that performance is constrained by memory bandwidth which is saturated with a small number of threads, regardless of vectorization.

[ Parent | on HN ]