News for hackers. Stuff that matters.
Re:Tensor Product Attention Is All You Need by bbcc90 on Wednesday January 22, 2025 @02:12AM (#42789970)
(trying to move the critique beyond the title...)

When trying to deploy llms in with larger context windows constrained environments 2 things start to hurt: a) increased memory footprint for longer KV cache b) increased decode speed due to longer context window. this paper addresses a) only, which is useful, but we are still left with b) (right?)

Re:Tensor Product Attention Is All You Need by verdverm on Wednesday January 22, 2025 @02:29AM (#42790070)
The more meaningful contribution may be (section 3.4)

> These variants illustrate TPA’s versatility in balancing memory cost, computational overhead, and representation power. By choosing which dimensions (heads or tokens) remain contextual and adjusting ranks (RQ, RK, RV ), TPA unifies multiple existing attention mechanisms— such as MHA, MQA, and GQA—under one framework, while potentially reducing the KV cache size by an order of magnitude during autoregressive inference.

re: the title, it might be the true one if their proofs hold up

---

I'm now curious if the Element-wise Attention is All You Need preprint can be fit into this framework. Sadly my math is not currently up to the task. It appears to offer even better computational savings during both training and inference while maintaining accuracy, though only tested with a smaller model

https://arxiv.org/abs/2501.05730

Re:Tensor Product Attention Is All You Need by ashupadhi01 on Wednesday January 22, 2025 @09:02AM (#42792923)
Curious to know what mathematics you are comfortable with. If you are able to understand the papers you mentioned, you must belong to the 99 percentile.
Re:Tensor Product Attention Is All You Need by verdverm on Wednesday January 22, 2025 @09:24AM (#42793155)
I was never good at proof writing. I found group theory and algebra interesting, topology and analysis eluded me. It's just been a while since I did any serious math thinking
Re:Tensor Product Attention Is All You Need by hansvm on Wednesday January 22, 2025 @11:23AM (#42794455)
EA doesn't quite fit in the same umbrella. EA has a constant cache size (it's just another classical recurrent architecture inspired by approximating transformers), where this paper gives speedups to a variety of true attention mechanisms which still require caches to be proportional to the sequence length.
Re:Tensor Product Attention Is All You Need by verdverm on Wednesday January 22, 2025 @12:09PM (#42795011)
very succinct and insightful, thank you!
Re:Tensor Product Attention Is All You Need by llm_trw on Wednesday January 22, 2025 @05:16AM (#42791180)
It addresses b too since decompositions are always smaller than the original tensor. It's usually the case that memory access is also slower than matrix multiplications so this will be faster. Burning flops to save memory movement.
Re:Tensor Product Attention Is All You Need by menaerus on Wednesday January 22, 2025 @09:12AM (#42793033)
> It's usually the case that memory access is also slower than matrix multiplications so this will be faster. Burning flops to save memory movement.

I haven't read this paper (yet) but isn't this the case that mostly applies to training and not so much to inference? A good example would be flash-attention, it trades the higher flops for better memory utilization but it's mostly irrelevant in inference workloads.

Re:Tensor Product Attention Is All You Need by verdverm on Wednesday January 22, 2025 @09:29AM (#42793199)
They claim an inference time savings to the kv cache
Re:Tensor Product Attention Is All You Need by menaerus on Wednesday January 22, 2025 @09:54AM (#42793451)
I skimmed through the paper real quickly. There's no performance data on inference speedups in the paper. Only the benchmarks relevant for training.

They also, interestingly, don't compare against the flash-attention. Flash-attention outperforms all of the other attention mechanisms mentioned in the paper: MHA, MQA, GQA, and MLA.

Re:Tensor Product Attention Is All You Need by verdverm on Wednesday January 22, 2025 @10:15AM (#42793686)
They aren't claiming speedups, they are claiming up to an order of magnitude less space needed for the kv cache at runtime. This translates to a smaller GPU or longer sequences in the same GPU
Re:Tensor Product Attention Is All You Need by menaerus on Wednesday January 22, 2025 @11:29AM (#42794534)
Under what circumstances can you cut down your LOADS and STORE from and to main memory by an order of magnitude without observing major improvements in algorithm runtime that is memory-bound?
Re:Tensor Product Attention Is All You Need by verdverm on Wednesday January 22, 2025 @12:08PM (#42794989)
AI models are compute bound, it's why we use GPUs
Re:Tensor Product Attention Is All You Need by menaerus on Wednesday January 22, 2025 @12:24PM (#42795217)
Incorrect. Self-attention is a highly parallel algorithm that makes it a great candidate for being a memory-bound workload once you have enough compute.

Both datacenter grade CPUs and GPUs have enough compute to carry out the self-attention computation but it is only the latter that has enough hi-bandwidth memory to make the algorithm really perform. If this hadn't been the case, the theory behind flash-attention wouldn't materialize, and it does, and reason being that (main) memory is slow.

Deep FFWD networks OTOH are compute-bound.

Re:Tensor Product Attention Is All You Need by zaptrem on Wednesday January 22, 2025 @10:30PM (#42800403)
Transformers are deep feedforward networks that happen to also have attention. Causal LMs are super memory bound during inference due to kv caching as all of those linear layers need to be loaded onto the core to transform only a single token per step.
Re:Tensor Product Attention Is All You Need by menaerus on Thursday January 23, 2025 @03:30AM (#42801966)
And I said something else?
Re:Tensor Product Attention Is All You Need by lostmsu on Friday January 24, 2025 @01:16AM (#42810916)
Memory bound only applies to low batch size scenarios AFAIK
Re:Tensor Product Attention Is All You Need by menaerus on Friday January 24, 2025 @02:17AM (#42811177)
This obviously depends on the hardware and the shape of the LLM model itself but, generally speaking, it's quite the opposite. The idea of batching is to grow the compute bandwidth per single request, bigger batch sizes with much more compute will put more stress to the underlying (cache, RAM) memory subsystem, no?

For N self-attention layers, there will be N compute (tensor) units doing the computation in parallel. To retire the computation, each compute unit will need to LOAD/STORE from and to the chip memory. At batch size B, this only becomes a bigger scale, e.g. B * (N, LOAD/STORE).

Re:Tensor Product Attention Is All You Need by lostmsu on Friday January 24, 2025 @10:19AM (#42813906)
If you have a batch of size 1, for every token you need to load the entire model from memory into cache as you go through it. If it is 32 you can produce 32 tokens while doing the same amount of loading from VRAM.
Re:Tensor Product Attention Is All You Need by menaerus on Friday January 24, 2025 @12:43PM (#42815493)
That's not how it works because if what you're saying had been true then the self-attention memory complexity would be O(1), e.g. regardless of the batch size. This obviously isn't the case since each batch computation necessitates it's own load/store memory bandwidth. I suggest reading one of the transformers papers to really understand how it works.
Re:Tensor Product Attention Is All You Need by lostmsu on Friday January 24, 2025 @02:04PM (#42816067)
This was a simplification. Of course you need some extra VRAM I/O based on your KV cache size.

But assuming your KV cache size is << model size, that simplification is pretty accurate.

See, e.g. https://www.databricks.com/blog/llm-inference-performance-en...

You can just scroll to the first chart they have that explains the idea.

Re:Tensor Product Attention Is All You Need by apophis-ren on Sunday February 02, 2025 @10:45PM (#42914705)
Flash attention is an implementation trick; you can implement MHA/GQA, for example, with flash attention.
Re:Tensor Product Attention Is All You Need by wseqyrku on Wednesday January 22, 2025 @05:54AM (#42791408)
> (trying to move the critique beyond the title...)

This is kind of a theme in HN now. The top comments are completely besides the point of the article/story/etc.

Re:Tensor Product Attention Is All You Need by msoad on Wednesday January 22, 2025 @08:42AM (#42792708)
I know. It is sad. Naming can also be seen as a way of showing respect to a hugely impactful paper if you want to be positive about it.