Re:AMD Open-Source 1B OLMo Language Models
by anon291 on Saturday November 02, 2024 @12:56PM
(#42027565)
> outside of trade show demos, kernels are condegened. What is true is there are recurring "themes/patterns" that are handled by engineers for a class of products. Lately this is flash attention
I never said they weren't using code generation. I said that each one requires a manual tune. You will set various parameters, determine if the generated code does well enough and then if there's performance to squeeze out, you modify the code generator.
> I guess you work at AMD.
Close but not quite
Re:AMD Open-Source 1B OLMo Language Models
by almostgotcaught on Saturday November 02, 2024 @12:58PM
(#42027581)
> that each one requires a manual tune
Ya definitely not - everyone uses grid search or whatever latest BPO tuning strategy.
Re:AMD Open-Source 1B OLMo Language Models
by anon291 on Saturday November 02, 2024 @01:17PM
(#42027700)
Oh right. Those require no engineering effort because I said so.
Re:AMD Open-Source 1B OLMo Language Models
by almostgotcaught on Saturday November 02, 2024 @01:19PM
(#42027720)
They require one person or team to engineer and then a whole bunch of people to use...? That doesn't resemble in the least what you were describing where each kernel is hand-tuned for each shape and device. But please do continue to insist you're still somehow right
Re:AMD Open-Source 1B OLMo Language Models
by anon291 on Saturday November 02, 2024 @06:15PM
(#42029600)
Sigh. I use to be an engineering manager for a kernels team. I think I know what I'm talking about. Yes, each kernel is paid individual attention to even if many are basically the same and require little rework. It's a lot of work. Now I work as an ic in the same field . I don't need to insist I'm right because it's what I do all day
Re:AMD Open-Source 1B OLMo Language Models
by almostgotcaught on Saturday November 02, 2024 @09:40PM
(#42030594)
> I think I know what I'm talking about
I've refuted your claims point by point and all you can say is "trust me bro"? Cool but I think you do not in fact know what you're talking about.
Re:AMD Open-Source 1B OLMo Language Models
by anon291 on Sunday November 03, 2024 @12:59AM
(#42031253)
Your claim is that there are automated methods (which I mentioned in my original post) to manage the complexity. My claim is that it requires a large team of engineers working on it. I'm not really sure what you think you've refuted.
Re:AMD Open-Source 1B OLMo Language Models
by lumost on Monday November 04, 2024 @05:26PM
(#42046634)
I work in DB kernels, everything gets hand tuned as there is economic reason for hand tuning it. The expectation in many of these systems is that there are no wasted cycles. You can codegen a decent kernel, but then someone will find a better approach - do you want the slower version of the product, or the faster one?
You can see this in action with matrix math libraries, folks have been hand tuning those for decades at this point.
Lol that's absolutely not true. What you're describing is literally impossible for any company that has more than one product family on the market since each product has different scratch sizes, number of vector registers, data types supported/emulated etc.
Outside of trade show demos, kernels are condegened. What is true is there are recurring "themes/patterns" that are handled by engineers for a class of products. Lately this is flash attention...
> Again, it is trivial to write a correct opencl matrix multiply, but that's never going to be the highest performance.
I guess you work at AMD. The reason AMD ships a whole bunch of binary kernels is not because someone tuned/designed each one but because AMD doesn't have a PTX/SASS equivalent. So each kernel has to be compiled at build time for each device (it's also why they can't have LTS support for architectures).