lysozyme

It’s interesting how Egypt’s efforts to monitor and test for malaria contributed to this accomplishment. It underscores how eradicating many infectious diseases will require a deep understanding not only of the disease itself, but also the cycles of transmission and the complex ecology of different hosts.

Malaria’s complex lifecycle [1] seems like it would be easy to “break” with different interventions, but we’ve seen historically malaria has been difficult to eradicate. Why is this?

1. https://en.m.wikipedia.org/wiki/Plasmodium#/media/File%3ALif...

For those like myself who design proteins for a living, and have had a lot of success doing it, the open secret is that well before AlphaFold, it was pretty much possible to get a good-enough structure of any particular protein you really cared about (from say 2005) by other means, namely Baker’s Rosetta.

I constantly use AlphaFold structures today [1]. And AlphaFold is fantastic. But it only replaces one small step in solving any real-world problem involving proteins such as designing a safe, therapeutic protein binder to interrupt cancer-associated protein-protein interactions or designing an enzyme to degrade PFAS.

I think the primary achievement is that it gets protein structures in front of a lot more smart eyes, and for a lot more proteins. For “everyone else” who never needed to master computational protein structure prediction workflows before, they now have easy access to the rich, function-determinative structural information they need to understand and solve their problem.

1. https://alexcarlin.bearblog.dev/multistate-protein-design-wi...

Agreed, the UX of a magnet timer on the fridge beats using any kind of smart device for the task of setting kitchen timers. Most of them start a simple count up if not programmed for a specific duration, so you can watch the seconds.

Most magnet timers also remember the last duration, so if you use the same timer a lot for the same task (tea, for example), it’s literally a single button press. The same operation on any kind of smart device contains a staggering number of steps, each of which requires cognition and attention.

Magnet timers are also super cheap, so you can get another one if you have two favorite durations. A simple solution meets a simple problem

Agreed on the hopes that these methods lead to novel biocatalysts (but they aren’t quite there yet).

David Baker’s lab has recently published on using their own diffusion model (RFdiffusion) to design novel biocatalysts that perform hydrolysis using a catalytic triad of serine, aspartic acid, and histidine, as well as an oxyanion hole, which is much more complex than the binders designed by AlphaProteo [1].

It gives me hope that we’ll soon be able to design biocatalysts as good as natural ones, but for any problem we care about.

1. https://alexcarlin.bearblog.dev/novel-enzymes-from-a-diffusi...

Good point! And a related topic, we call the organism that lives happily in our gut E. coli and we also call the organisms that cause disease the same name. What’s the difference?

It turns out that the Escherichia coli (to spell out its Latin binomial) that cause disease are in some sense “diseased” themselves: the genes that enable them to be pathogenic, or make them pathogenic, I should say, are originally from a phage, a type of virus that infects bacteria [1]. In a manner that is not the same as, but conceptually similar to how HIV inserts its genes into the human’s genome, phages insert their genes (termed the “prophage”) into the bacterial genome.

In addition, most strains of pathogenic Escherichia are also holding on to an entirely separate, small, circular “genome” called a plasmid, also of exogenous origin, that contains additional genes that make them pathogenic.

So in addition to wide genome variation within the “species” (which is not really the same thing for bacteria as for mammals, mind you) of Escherichia coli, many subtypes have additional genetic material from endogenous sources that substantially changes their observed characteristics (phenotype).

1. https://en.m.wikipedia.org/wiki/Escherichia_coli_O157:H7

The flip side of this is that progress in ML for biology is always going to be _slower_ than progress in ML for natural languages and images [1].

Humans are natural machines capable of sensing and verifying the correctness of a piece of text or an image in milliseconds. So if you have a model that generates text or images, it’s trivial to see if they’re any good. Whereas for biology, the time to validate a model’s output is measured more in weeks. If you generate a new backbone with RFDiffusion, and then generate some protein sequences with LigandMPNN, and then want to see if they fold correctly … that takes a week. Every time. Use ML to solve _that_ problem and you’ll be rich.

TFA mentions the difficulty of performing biological assays at scale, and there are numerous other challenges. Such as the number of different kinds of assays required to get the multimodal data needed to train the latest models like ESM-3 (which is multimodal, in this context meaning primary sequence, secondary structure, tertiary structure, as well as several other tracks). You can’t just scale a fluorescent product plate reader assay to get the data you need. We need sequencing tech, functional assays, protein-protein interaction assays, X-ray crystallography, and a dozen others, all at scale.

What I’d love to see companies like A-Alpha and Gordian and others do is see if they can use the ML to improve the wet lab tech. Make the assays better, faster, cheaper with ML. Like how they use ML to translate the electrical signals of DNA passing through the pore into a sequence in the Nanopore sequencers. So many companies have these sweet assays that are very good. In my opinion, if we want transformative progress in biology, we should spend less time fitting the same data with different models, and spend more time improving and scaling wet lab assays using ML. Can we use ML to make the assay better, make our processes better, to improve the amount and quality of data we generate? The thesis of TFA (and experience) suggests that using the data will be the easy part

1. https://alexcarlin.bearblog.dev/why-is-progress-slow-in-gene...

Probably worth mentioning that David Baker’s lab released a similar model (predicts protein structure along with bound DNA and ligands), just a couple of months ago, and it is open source [1].

It’s also worth remembering that it was David Baker who originally came up with the idea of extending AlphaFold from predicting just proteins to predicting ligands as well [2].

1. https://github.com/baker-laboratory/RoseTTAFold-All-Atom

2. https://alexcarlin.bearblog.dev/generalized/

Unlike AlphaFold 3, which predicts only a small, preselected subset of ligands, RosettaFold All Atom predicts a much wider range of small molecules. While I am certain that neither network is up to the task of designing an enzyme, these are exciting steps.

One of the more exciting aspects of the RosettaFold paper is that they train the model for predicting structures, but then also use the structure predicting model as the denoising model in a diffusion process, enabling them to actually design new functional proteins. Presumably, DeepMind is working on this problem as well.

As cool as this is, the word “microplastics” is a little misleading. There are dozens of types of plastic in common use, each made from a different monomer with a different chemical linkage, of which PET is only one. The engineered protein in TFA will only work on PET and we’ll need to design new proteins for the other types of plastic. (I can help with that.)

The problem with enzymes eating plastic is that enzymes are small Pacman-shaped protein blobs that are maybe 10 nanometers in diameter, whereas things made of plastic like bottles or even microplastics are huge in comparison. How do you get the little Pacman jaws around the bottle to start breaking it down?

The research paper [1] describes the authors’ effective innovation. They make a protein where one end is a pore-forming shape, and the other end is a PET cutting (called a PETase in the jargon of the field). This way, their protein can access nooks and crannies in the macroplastic shapes, allowing tons of copies of this small enzyme to fully degrade a bottle.

Without this, a great deal of physical agitation is required to break down the plastics into small enough chunks that earlier Pacman enzymes could work on, increasing the time and the cost.

I hope we’ll see the idea of linking the enzymatic “scissors” to a protein pore be used to engineer enzymes to degrade other types of plastics in the future, as the general idea of getting the catalytic machinery into physical contact with every bit of the bottle is broadly applicable to all plastics, not just PET (which is great news)

1. https://phys.org/news/2023-10-scientists-artificial-protein-...

Great article. I think we find time and time again that the best leaders are really “cheerleaders”. It turns out that inspiring people, personally living the values you espouse, and giving your team trust and respect is the key to success.

In my experience, people excel in environments where they are given a high level of trust, autonomy, and clear goals on the timescale of months and years.

Teams on the rack of egotistical middle managers get stretched thin until they break. There aren’t a lot of good examples of “micromanagitis” being cured. So-called leaders taking credit for their team’s work leads to their team not trusting that leadership has their growth in mind. Art Blakey’s real legacy is both the amazing players he mentored and tutored, but also the attitude that if you give people the tools they need, they’ll excel

I think parent has hit on the how and GP has hit on the why.

How LLMs are able to give convincing wrong answers: they “can predict the correct ‘shape’ of an answer” (parent).

Why LLMs are able to give convincing wrong answers is a little more complicated, but basically it’s because the model is tuned by human feedback. The reinforcement learning from human feedback (RLHF) that is used to tune LLM products like ChatGPT is a system based on human ranking. It’s a matter of getting exactly what you ask for.

If you tune a model by having humans rank the outputs, despite your best efforts to instruct the humans to be dispassionate and select which outputs are most convincing/best/most informative, I think what you’ll get is a bias towards answers humans like. Not every human will know every answer, so sometimes they’ll select one that’s wrong but likable. And that’s what’s used to tune the model.

You might be able to improve this with curated training data (maybe something a little more robust than having graders grade each other). I don’t know if it’s entirely fixable though.

The brilliant thing about the parent’s comment about the “shape” of the answer is that it reveals how much humans have (uh, historically, now, I guess) relied on the shape of information to convey its trustworthiness. Expand the notion of “shape” a bit to include the medium. If somebody bothered to take the time to correctly shape an answer, we take that as a sign of trustworthiness, like how you might trust something written in a carefully-typeset book more than this comment.

Surely no one would take the time to write a whole book on a topic they know nothing about. Implies books are trustworthy. Look at all the effort that went in. Proof of effort. When perfectly-shaped answers in exactly the form you expected are presented in a friendly way and commercial context, they certainly read as trustworthy as Campbell’s soup cans. But LLMs can generate books worth of nonsense in exactly the right shapes without effort, so we as readers can no longer use the shape of an answer to hint at its trustworthiness.

So maybe the answer is just to train on books only, because they are the highest quality source of training data. And carefully select and accredit the tuning data, so the model only knows the truth. It’s a data problem, not a model problem

When evaluating this work, it’s important to remember that the functional labels and protein family assignments on each of the 280 million input sequences were originally assigned by an HMM model using human curated sequence groups as part of the pfam project, so the model is predicting a prediction (or perhaps conditioned on a prediction would be more accurate).

Furthermore, the authors must engage a lot of human curation to ensure the sequences they generate are active. First, they pick an easy target. Second, they employ by-hand classical bioinformatics techniques on their predicted sequences after they are generated. For example, they manually align them and select those which contain specific important amino acids at specific positions which are present in 100% of functional proteins of that class, and are required for function. This is all done by a human bioinformatics expert (or automated) before they test the generated sequences. This is the protein equivalent of cherry-picking great examples of, for example, ChatGPT responses and presenting them as if the model only made predictions like that.

One other comment, in protein science, a sequence with 40% identity to another sequence is not “very different” if it is homologous. Since this model is essentially generating homologs from a particular class, it’s no surprise at a pairwise amino acid level, the generated sequences have this degree of similarity. Take proteins in any functional family and compare them. They will have the same overall 3-D structure—called their “fold”—yet have pairwise sequence identities much lower than 30–40%. This “degeneracy”, the notion that there are many diverse sequences that all fold into the same shape, is both a fundamental empirical observation in protein science as well as a grounded physical theory.

Not to be negative. I really enjoyed reading this paper and I think the work is important. Some related work by Meta AI is the ESM series of models [1] trained on the same data (the UniProt dataset [2]).

One thing I wonder is about the vocabulary size of this model. The number of tokens is 26 for the 20 amino acids and some extras, whereas for a LLM like Meta’s LLaMa the vocab size is 32,000. I wonder how that changes training and inference, and how we can adopt the transformer architecture for this scenario.

1. https://github.com/facebookresearch/esm

2. https://www.uniprot.org/help/downloads

Nest still sends me a monthly summary email for account that’s maybe nine years stale, despite my repeated attempts to get them to stop. Their customer support says there’s “nothing they can do”.

The Nest smoke alarms at a friend’s house say (as in play over the speakers a voice saying) “Remote access attempt detected” every so often while we’re sitting in the living room. At least twice a year they go off in the middle of the night (without smoke).

So I guess when you’re contacting customers who don’t want it, and your alarms wrongly sound, I would say you have a problem with your false positive rate

And quite a bit longer than that even for the human brain to convolve safely with its surroundings and with other human brains.

One yet further objection to the many excellent already-made points: the deployment of LLMs as clean-slate isolated instances is another qualitative difference. The human brain and its sensory and control systems, and the mind, all coevolved with many other working instances, grounded in physical reality. Among other humans. What we might call “society”. Learning to function in society has got to be the most rigorous training for prompt injection I can think of. I wonder how a LLM’s know-it-all behavior works in a societal context? Are LLMs fun at parties?