rvanbruggen

Re:Introduction to Flash Attention – improving the efficiency of LLMs on Wednesday April 10, 2024 @06:22AM (#39989053)

Super interesting concept. Thanks for sharing.

Re:What Is Rotary Position Embeddings (Rope) Scaling? on Tuesday April 02, 2024 @09:53AM (#39905789)

LLMs rely on Rotary Position Embeddings (RoPE) to understand the relative position of words within a sequence. Each word in the sequence is assigned a unique embedding based on its position. This embedding is calculated using a combination of sine and cosine functions, incorporating its distance from the beginning and end of the sequence. However, standard RoPE struggles with longer sequences than those encountered during training.

What Is Rotary Position Embeddings (Rope) Scaling?

Posted by rvanbruggen on Tuesday April 02, 2024 @09:53AM
from the hopsworks.ai dept.

https://www.hopsworks.ai/dictionary/rope-scaling

( Read More... | 1 comment | 5 points | on HN )

What Is Sample Packing?

Posted by rvanbruggen on Monday March 04, 2024 @03:56AM
from the hopsworks.ai dept.

https://www.hopsworks.ai/dictionary/sample-packing

( Read More... | 1 comment | on HN )

Re:What Is Sample Packing? on Monday March 04, 2024 @03:56AM (#39588359)

Interesting - did not know about this term yet. Not surprising as this field is evolving so quickly!

The 3 Why's of feature stores for ML and AI

Posted by rvanbruggen on Tuesday February 27, 2024 @06:11AM
from the blog.bruggen.com dept.

https://blog.bruggen.com/2024/02/the-3-whys-of-feature-stores-for.html

( Read More... | 1 comment | 3 points | on HN )

Re:The 3 Why's of feature stores for ML and AI on Tuesday February 27, 2024 @06:11AM (#39522684)

Some good content to think through 1. Do I need a feature store? 2. Should I build or buy one? 3. Is Hopsworks a suitable candidate?