ml_basics

The untimely death of David MacKay was a tragic loss for the scientific community. It's astonishing that he wrote the textbook on information theory for the ML community as well as his contributions to rational discussion about energy policy.

He had a direct personal impact on my life since my introduction to ML came from video recordings of his lectures on ML and information theory. I am grateful to have learned from such an excellent teacher. Like a Feynman of our times.

Read about half of the article wondering when it was going to explain why carplay is addictive

January 2026 when i started using opus 4.5 and understood that it could do actual useful work beyond coding small snippets

maybe we will look back and also think that the current slate of IPOs were reasonable valuations?

this will change as inference demand increases (which is happening right now faster than many people expected)

During my time at university studying pure mathematics I had an interesting experience of doing a challenging sheet of combinatorics problems during a vacation. Every day I attempted one question and got stuck on it. Then the next morning I woke up knowing the solution. It was a recurring thing: this happened every day for about 2 weeks until I had solved all the problems.

For me this a big eye opener about the importance of sleep and relaxed thinking to solve challenging problems.

how much of the cloud revenue is from Anthropic revenue sharing?

They are referring to a thing called "speculative decoding" I think.

This is from 11th September

The graph at the beginning showing the cost of sequencing over time falling faster than Moore's law stops in 2015. Would love to see how things have progressed since then. Casually googling i only saw plots up to 2021 but looks to me like progress is now slower than Moore's law since ~2015. Maybe things will change when Nanopore gets more reliable

what's the relationship between this work and the recently announced voice models from Microsoft AI? https://microsoft.ai/news/two-new-in-house-models/

It's a europeanism. In both French and German (and probably other EU languages) the word for "subsidy" is something like "subvention" so native speakers of these language often reach for an unnatural word in English.

Btw other examples include "actually" which is used to mean "currently", and "eventually" which is used to mean "maybe".

Personally I'm torn whether to consider this incorrect use of the language as it is quite widespread. Maybe it would be better to consider this as the emergence of a new dialect.

And American tech companies have been substantially more successful that European tech companies so maybe it does actually make sense to have stock options as part of compensation rather than employer bonuses?

It's quite remarkable how much the goal posts have shifted when it comes to what is impressive with AI/ML. Things like this are a good reminder.

10 years ago the GAN paper came out and everyone was excited how amazing the generated image quality was (https://arxiv.org/abs/1406.2661)

The amount of progress we've made is mind boggling.

To be fair to the authors they are affiliated with a university and not a big industrial lab, so they may be working with significantly constrained resources. Not sure exactly what the best solution is for this case given that it affects most people outside of a very select few.

Cool paper. Really interesting to see how even quite straightforward architectural modifications haven't yet all been exhausted yet, despite all the resources being poured into LLMs

They make the yoghurt, then pasteurise it (I guess so it has a longer shelf life). So it tastes roughly like yoghurt but doesn't have any of the good bacteria. I've also seen that sometimes lactobacteria are then artificially added back in so that they are present but in a controlled way.

> As of July 3, 2023, we’ve disabled the Browse with Bing beta feature out of an abundance of caution while we fix this in order to do right by content owners. We are working to bring the beta back as quickly as possible, and appreciate your understanding!

Demo website with speech-speech translation examples https://google-research.github.io/seanet/audiopalm/examples/

> We introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2.

Direct link to demo video showing speech-to-speech translation: https://google-research.github.io/seanet/audiopalm/examples/... (see website for more example)

I was thinking atomic as in "atomic operations"

bfloat16 is probably familiar only to ML practitioners; it's a reduced precision floating point format designed for ML models. I was surprised to learn that the "b" stands for "brain", as in the team at google that developed it along with many other advances in machine learning.