News for hackers. Stuff that matters.

A comment on Tencent's 'Hunyuan-T1'–The First Mamba-Powered Ultra-Large Model. Parent comment

Re:Tencent's 'Hunyuan-T1'–The First Mamba-Powered Ultra-Larg... by gardnr on Sunday March 23, 2025 @02:25AM (#43451202)
There was a paper[1] from last year where the authors discovered getting the model to output anything during times of uncertainty, improved the generations overall. If all of the post-training alignment reasoning starts with the same tokens then I could see how it would condition the model to continue the reasoning phase.

1: https://arxiv.org/abs/2404.15758

[ Parent | on HN ]
Re:Tencent's 'Hunyuan-T1'–The First Mamba-Powered Ultra-Larg... by throwawaymaths on Sunday March 23, 2025 @07:20AM (#43452223)
this is probably because the thinking tokens have the opportunity to store higher level/summarized contextual reasoning (lookup table based associations) in those token's KV caches. so an "Ok so" in position X may contain summarization vibes that are distinct from that in position Y.
[ Parent | on HN ]