News for hackers. Stuff that matters.
Re:Tighter bounds on the expressivity of transformer encoders by eutectic on Tuesday April 25, 2023 @01:57PM (#35704209)
I think it means the analysis accounts for rounding error.
Re:Tighter bounds on the expressivity of transformer encoders by ftxbro on Tuesday April 25, 2023 @03:55PM (#35705806)
Yes it's this. They have identified an expressivity level that they are calling FOC[+;MOD] which is intuitively supposed to capture the expressivity that transformers and transformers-with-rounding have in common. More formally they are trying to track down exactly how this level compares to that of transformers, transformers-with-rounding, simplified stateless counter machines, and uniform TC(0), and they have done it with partial success and they are reporting their results.