sagex

Isn't the network too shallow to be challenging to decode? One can easily produce few layer network to solve specific goal with construction techniques, so not a surprising result in my view.

With open models delivering close to frontier level capability, I really don't see how these labs can only focus on having best model to sustain their business. Although these model's usage will grow, I think and hope that the serving will be distributed among many players.

Essentially, if SOTA models are freely available, the playing field then is on inference where people from all around the world can participate. This basically dilutes the value of model trainer like Anthropic. I can see the panic.

Don't know why are they even pursuing Gemini. Just download the Kimi, call it Kimini and serve it on your GPU. Maybe then train next architecture based on this!

I believe invention of Transformers and especially Attention mechanism do have influence from past research but its not definitely only the Schmidhuber's work. Said that, if we remove the papers mentioned by Schmidhuber from history, I am quite certain that there will be no influence in the discovery of Transformers, hence his works can not be the root. He has to grow up and accept that work and equations can appear similar, looking at inverse squared law and saying Newton stole that from someone is being dishonest.