lucrbvi

no

It means that modern neural networks are way more interpretable than older ones. We can understand how they work way more easily than earlier ones.

Space Bunny Alpha is probably MiniMax M3.1 (rumors on Twitter since it seems to have a similar tokenizer).

There are a lot of open-research on pre-training, post-training and RL data mixtures and sourcing.

I recommend checking papers from Datalogy, Nvidia Nemotron, Ai2 (Ollmo, Tulu, ...) and the recent model from Aleph Alpha if you want to learn more.

They only care because open-weight helps to drive the GPU business notably thanks to inference providers (Baseten, Together, Mistral, ...).

At least their support helps the open-weight ecosystem.

I think one of the best ways of stopping lock-in is to be file-based. Obsidian is a good example of this. Users can just keep their data as is and move on with their lives.

You should call Jev-like models when you give it a JSON-like structure to produce, it is useful when you need _some_ intelligence in your code.

They are using it to evaluate checkpoints during the training, they are probably not using the benchmarks for training the models. It's a common practice for big reinforcement learning runs.

> This is why it seems that alignment for a generalized, chat-style AI is a very hard problem, perhaps impossible. You can't align it to solve a certain kind of problem and keep it general to any question. The two goals are in conflict with each other.

I think this problem is going to be solved soon (hopefully), check out for the Persimmon model[0] from Humans&. They train it to mimic humans, it may seems bad but it could be _really_ useful to train an AI to be aligned to humans and understand their goals really well since they can use Persimmon to create a "fake human" following a defined goal that their big AI model will learn to estimate.

It's still early but I think this is what they are heading toward.

[0]: https://persimmon.humansand.ai/blog/persimmon.html

That's a lot of ®, curious how ClickHouse® Inc. is treating the use of its name by others ... Hopes it's not like Oracle with JavaScript

The target model is the original LLM that is large and expensive. It can verify candidate tokens in a single forward pass. It means you give all the context + candidate tokens that passes in parallel in the backbone, then you pass the language head (a matmul transformation to produce the token distribution) on all the candidate tokens and you can keep or drop tokens based on how many "quality" you want.

Under the EU's AI Act non-profits and universities can just bypass these constents to make non-commercial datasets btw

It would be fun to train a small vision neural network to recognize ads instead of relying on HTML or network requests. I don't know if anyone has done this yet.

Mistral has the capability of training such models. Take a look at Poolside[1], they are claiming to pre-train their Laguna series of models on 4,096 NVIDIA H200 GPUs[2]. Mistral has approximately 13,800 NVIDIA GB300 GPUs, which are nearly 2x more efficient for training.

The problem with Mistral is that they do not seem to have aligned incentives to train big open-weight models, even if the teams would like to.

[1]: https://poolside.ai/ [2]: https://poolside.ai/blog/introducing-laguna-s-2-1

"world model" is, sadly, a new buzzword; most of the world models we are seeing are video generative models (like GPT-Image), they receive an action vector and previous frames and must generate the next frame. It's working of course.

The whole field of world models also have lighter approaches based on using latent representations as the target (and not pixel frames). LeWorldModel[0] and DINO-WM[1] are a good way to learn how these works.

Strangely, most of the startups in the space of world models are focused on video-generative world models.

A good application of world models is in robotics where an action policy generate a distribution of actions, you can then predict the future state of the world and use some reward policies to choose the "best" future state.

Seems like MAI models from Microsoft are not going to be open-weight soon, but they are sharing a lot of details in the making of these models, which is a weird position.

Such a gem, thanks to the author for sharing it's findings :)

The only problem I have with planing in latent space is that it can be really noisy and not representative of the positions in the game (the latent are trained for semantic, so the optimizer can focus a set of specific features and can skip positions, which means it cannot know "where" to go by optimizing on the latents directly).

Anthropic theorize that middle layers in an LLM is a "J-Space" used to "think" about the future answer or about abstract concepts.

Their method is used to identify which tokens can appears in which layers of the model.

In the article the author says they are doing reinforcement learning with LLMs.

How does this differ from dottxt's Outlines[0] on the technical level? Are you using some JSON grammar to force the LM head distribution to follow it?

[0]: https://github.com/dottxt-ai/outlines

it's using webgpu kernels, probably a false positive

I really like QBE but I hope they will make it a true library because I don't want to launch a subprocess to compile a program.

Cool but why would anyone write an agent in C? Since the language is verbose it seems to be a painful choice to use C for this task.

They have added a lot of optimization focussing on the KV-cache, so they can have a much larger window without eating all the VRAM.

The 1M window might be usable, but it will probably underperform against a smaller window of course.

Mollie seems to only provide services to business based in European Economic Area, Switzerland and the UK [0], so yes?

[0]: https://help.mollie.com/hc/en-us/articles/115002116105-Can-I...