JaydenTeoh

this is not true. even with ideas with provable/strong scalability, there was still a lack of resources (especially GPUs) to validate such ideas at scale. ultimately, "showing the money" still requires some initial faith from those higher up who control the resources.

hey thanks for the comment, author here. That's a fair read of the loss numbers, I could only fund up to 1.2B scale myself and I don't want to oversell the results. But note that the benefits that Soft-MTP delivers in the paper is *consistently growing across scale*. The residual is irreducible with scale and will only become noisier as model grows bigger, so removing it via Soft-MTP is intuitive. It is also easily verifiable cos it only needs minor code change, codebase: https://github.com/JaydenTeoh/Soft-MTP

there is reproduction of Soft-MTP's benefit over standard MTP at 35B scale but it is only one data point for now: https://x.com/HivazeMe/status/2118432486724391217