desideratum

Yes my findings and thoughts were pretty much identical. I actually think you can get something reasonable at 1.3B params with the correct training recipe, but definitely not at this compute/token budget.

One thing I found was that the model would pretty much always emit solutions from its training data when asked to solve problems, but it was much better at using Bash commands to explore a codebase, for example.

The Hugging Face folks have a great post on also using CAI for more vibes/character post-training than harmlessness https://huggingface.co/blog/constitutional_ai#oh-honey-lets-...

This is a gross simplification of the process - you would typically use order(s) of magnitude more data and compute, and a substantial amount of online reinforcement learning to elicit emergent tool use capabilities.

Many recent OSS models have great tech reports where you can learn more about these kind of things: Kimi 2.5 https://github.com/MoonshotAI/Kimi-K2.5/blob/master/tech_rep... GLM 5 https://arxiv.org/abs/2602.15763 DeepSeek R1 https://arxiv.org/pdf/2501.12948

I appreciate the kind words very much : )

I see what you mean, but I disagree. I expect that Claude Code is backed by a separate post-train of Claude base which has been trained using the Claude Code harness and toolset.

Oh I wouldn't be surprised. This is a sample from one of the OSS code datasets I'd used, which are all generated synthetically using LLMs. Data is indeed the moat.

This is a great question. You definitely aren't training this to use it, you're training it to understand how things work. It's an educational project, if you're interested in experimenting with things like distributed training techniques in JAX, or preference optimisation, this gives you a minimal and hackable library to build on.

Oh, and if you want to utilize 120Hz on the XDR display, you're going to have to replace your perfectly functioning Mac.

> Mac models with M1, M1 Pro, M1 Max, M1 Ultra, M2, and M3 support Studio Display XDR at up to 60Hz. All other Studio Display XDR features are supported.

It's mind-boggling that Apple is considering the base 27 inch Studio Display with the same 4 year old panel, but with some new accessories slapped on an "upgrade".

Thanks for sharing this. I agree w.r.t. XLA. I've been moving to JAX after many years of using torch and XLA is kind of magic. I think torch.compile has quite a lot of catching up to do.

> XLA isn't at present particularly useful at scheduling across machines,

I'm not sure if you mean compiler-based distributed optimizations, but JAX does this with XLA: https://docs.jax.dev/en/latest/notebooks/Distributed_arrays_...

The Scaling ML textbook also has an excellent section on TPUs. https://jax-ml.github.io/scaling-book/tpus/

Aside: this guy regularly posts on the Discord server for an open-source post-training framework I maintain, demanding repayment for bugs in nightly builds and generally abusing the maintainers.

This is an exceptional salary for the UK.

I'd reccomend checking out the CUDA mode Discord server! They also have a channel for Metal https://discord.gg/ZqckTYcv

torchtune (https://github.com/pytorch/torchtune) - a PyTorch library for fine-tuning LLMs, particularly for memory-constrained setups. Try it out and fine-tune Llama3.1 8B on a single RTX 4090!

Hi Stefano. I'm an ML Engineer/Researcher looking for a new role, and very interested in learning more about Epistemic. Am I correctly visualizing something similar to https://www.connectedpapers.com/ as part of your product? I can see incredible potential using ML/NLP to build knowledge graphs from a diverse set of sources of biomedical knowledge, from derisking future research to exposing connections and ideas people haven't even considered! Thanks, Salman

Nick Bostrom's "Superintelligence" is a sober perspective on this issue and a very worthwhile read.

Hi David, wonder if you'd be open to remote within the UK? I'd be interested in the ML Engineer role primarily, but would also happily be considered for other roles.

Some truly impressive results. I'll pick my usual point here when a fancy new (generative) model comes out, and I'm sure some of the other commenters have alluded to this. The examples shown are likely from a set of well-defined (read: lots of data, high bias) input classes for the model. What would be really interesting is how the model generalizes to /object concepts/ that have yet to be seen, and which have abstract relationships to the examples it has seen. Another commenter here mentioned "red square on green square" working, but "large cube on small cube", not working. Humans are able to infer and understand such abstract concepts with very few examples, and this is something AI isn't as close to as it might seem.

Thanks for this. I had the same thought about this being a lesson they'll need to learn. The other two engineers put up very little resistance and seem to be perfectly happy with their TC.