I doubt anyone is still looking at this thread but I did actually start playing with RWKV by adding sacrificial training techniques to it and the results look promising, at least for early training.