News for hackers. Stuff that matters.
Re:Implementation of mixture of experts language model in a ... by avisoori1x on Friday March 22, 2024 @01:22PM (#39792853)
Thanks! So this is something I tried and qualitatively I didn't see a huge difference. I'd like to swap out my hand rolled modules with standard pytorch modules for self attention etc. and train it on the wikipedia English split. That's on my to-do list for sure.

There are no comments on this comment yet.