Careful with anthropomorphising. Communication is much more than vocalisation of abstract thought (i.e. the way humans mostly use it).

Animals don't communicate the same way we do and it might be very misleading to associate concepts such as grammar with their utterances. While some species use distinct sounds to express certain situations (like "careful, there's an airborne predator" vs "careful, there's a snake"), not all "verbal" expressions serve such purpose.

The general consensus in contemporary science is that bird songs in particular are utilities to attract mates, mark territory, and the like; not exchanging information (other than things like "this is my nesting place" or "I'm healthy and available", but not in a sense of words or sentences, but more like the acoustic equivalent of gesturing).

"Careful with anthropomorphising."

The confabulation elements of this particular AI technology would be particularly concerning in this use case. This would be the most literal anthropomorphization you could imagine. Applying a strictly human language model to a non-human will inevitably result in humanization of the target by the nature of the bias of the system.

I fed "You are a rock. What might you like to communicate to human beings?" to ChatGPT and it answered just fine, off of nothing more than that input, giving one of its typically highly sanctimonious and high-handed lectures about environmentalism and managing to work in complaints about climate change, as naturally a rock would deeply care about. (It has now even titled my session "Rock's Wisdom To Humans". Good lord, ChatGPT has certainly become an annoying prick.) This has everything to do with the biases in the technology and nothing to do with the thing putatively doing the communication. Feeding it some sort of data from a real whale isn't going to do any better; it'll happily confabulate away.

"Bias" in this case has a more technical meaning, which refers to the representational capability of the system, not political biases intrinsically. It so happen that ChatGPT immediately leaps to political topics, which is doing a great job of obscuring the way I'm trying to use the term, but that's because of its own internal structures which have been tuned in that direction. This, in fact, accidentally serves as as good an example as anything of how highly tuned ChatGPT has become to give politically correct answers, given the political neutrality of the prompt. But what I'm referring to here is just that a human language model is going to be biased in human ways. It can't help but turn anything you feed it into a human perspective, so using it for trying to understand non-human perspectives and communication is going to be intrinsically a mistake.

I think OP was imagining using a LLM trained only on animal sound data, not using a LLM trained on human data to interpret animal data.

Which begs the question what the added benefit would be from such an approach, i.e. what could we even learn from that compared to other methods?

To quote from an article in The Royal Society:

> [...] bird song evolution can also be understood as a response to natural selection, as when oscine passerine species in urban habitats raise the frequency of their song as a means to overcome traffic noise.

Things like changes in the vocalisation can just be adjustments to the environment or simply due to sexual selection. How would an LLM be able to capture this, given that it would lack such additional information?

Then there's this quote from the same article:

> In contrast to the expectation of gradual change through cultural or genetic drift over time, the results instead demonstrate that plastic traits such as song can exhibit punctuated evolution, with bursts of trait divergence interrupting extended periods of stasis.

So even if a model was trained on (regional) bird dialects, the data could become obsolete basically over night, as the structure and patterns might change dramatically between breeding seasons. The researchers used statistical modelling (classical Gaussian mixture models) to analyse the data. I'm a bit lost as to how LLMs could be applied in this context (in that I have no idea what kind of scientific questions they could answer).

The full study can be found here: https://royalsocietypublishing.org/doi/10.1098/rspb.2021.206...

It's not unreasonable to expect that a big enough model could learn some kind of useful feature space for bird vocalizations. You are essentially replacing a parsimonious quasi-parametric model (Gaussian process) with an enormous fully nonparametric one. Transformer units should be as well-suited for this as for any other sequence learning task.

But the LLM would need to translate the sounds to human language, and that's where the issue would arise.

Yes, this was my logic. A model trained on bird song may make birds turn their heads when it babbles something in their frame of reference, but if you want to "use LLM" to make it human comprehensible you need a human LLM at some point, which will impose humanity on its input regardless of what you feed it, because all a human LLM can represent is human language. A human LLM can't help but make whatever it is used on human. A bird LLM would have the same effect; no matter what you feed it, you'll get something in the bird input's space.

Not sure you're really getting the idea. In this scenario, there wouldn't be human llm and dolphin llm, you would train human speech and dolphin communication in one model. sound tokens are sound tokens. pre-trained transformers don't discriminate.

The concern would be that it would gnerate a bunch of spurious correlations between the two. You would be mixing two kinds of datasets, where it’s not clear that the dolphin sounds are linguistic. It’s also not clear that LLMs would correctly map human language to a non-human one. Depends on your theory of meaning. Wittgenstein would say if a lion could talk, we would not understand it, because lmeaning is use, and lions would use language differently than humans.

> highly sanctimonious and high-handed lectures... Good lord, ChatGPT has certainly become an annoying prick.

You've hit the nail - or dare I say rock - on the head.

That said, I think the bias we've both encountered is an aspect of the model's initial instructions and not reflective of LLMs in general.

You might be interested in work being done by the Earth Species Project https://www.earthspecies.org/what-we-do/projects

That is fascinating, thank you! Self-supervised learning that could correlate animal sounds with behavior. Exactly the kind of thing I was thinking about.

Yes and theoretically if it trains on human language as well, you'd be able to translate between the two (assuming such a thing was possible, i.e assuming say dolphin communication was a complex language).

That's the part I don't quite understand -- I know ChatGPT can translate quite well between languages.

But I'm not clear on whether that's because it's been trained on a bunch of aligned translations and bilingual dictionaries and whatnot, or if it's detecting the same repeating patterns in languages and figuring out the translation mappings on its own.

Is it the latter? Could a ChatGPT actually translate something like dolphin communication into semi-meaningful English, that wouldn't just be entirely nonsense?

It's the latter.

That's why it can only be trained on 1.8% french by word count and speak fluent french. It's not that the amount of french that was in its corpus would have been enough to train it to that equivalent level alone (it wouldn't, not by a long shot). It's that there's positive transfer of knowledge. whatever it learns predicting english helps it learn other languages much faster. it's not the only display of positive transfer either. llms trained on code reason better.

It's also why the open ai instruct tuning dataset was almost entirely english yet it follows instruction in other languages just fine.

If there's something to be translated, a language model trained on both will translate it. it just needs to learn the concept of translation but the human side of the dataset should take care of that

Really i only say theoretically because of 2 things

one. first, a multimodal text-audio model that shows positive transfer between text and audio needs to be demonstrated. otherwise, this hypothetical human-dolphin model might need too much data to be feasible.

2. we're going to need a lot of dolphin speech audio.

Once one of these models are trained on a huge video corpus like YouTube, which includes lots of videos of animals, it might just be that stuff like this is only a prompt away, assuming there is enough training data.