News for hackers. Stuff that matters.

A comment on RWKV Language Model. Parent comment

Re:RWKV Language Model by Ey7NFZ3P0nzAe on Thursday January 02, 2025 @03:39AM (#42572781)
I'm quite interested in repeng [0] (representztion engineering) for steerability of (so fzr transformer based) LLMs and was wondering if anyone had tried such methods on rwkv (or mamba for that matter). Maybe there are some low hanging fruits about it.

[0] https://github.com/vgel/repeng/issues

[ Parent | on HN ]
Re:RWKV Language Model by pico_creator on Thursday January 02, 2025 @07:33AM (#42573878)
One of the interesting "new direction" for RWKV and Mamba (or any recurrent model), is the monitoring and manipulation of the state in between token. For steerability, alignment, etc =)

Not saying its a good or bad idea, but pointing out that having a fixed state in between has interesting applications in this space

[ Parent | on HN ]