I think the purpose of this experiment was to have some fun.

Indeed :)

> This will always be a big problem in AI

I agreed with everything you wrote except for "always".

Considering even actual humans often have disagreements about meanings of songs, I'm not sure how an AI is supposed to deduce the "correct" one.

I don't think that the imprecision of human song interpretation can be used as evidence that AIs are inaccurate.

Neither am I. But 10 years ago I would have said I had no idea how an AI was supposed to recognise objects in a picture, and now they do it better than humans. Tech moves fast.

Recognising objects in pictures is something that, in principle, a machine could do better than humans, because there is ground truth.

I think it's not possible in principle for a machine to do better than humans at recognising human emotions, because a human can't exactly be wrong at this. (Other than edge cases such as not knowing the right name for an emotion, perhaps.)

But suppose a sample of 1000 humans answered a survey on emotional content of a song and a machine was able to predict the distribution within some tolerance. In a sense, this would be superhuman performance, but even if we don't like that interpretation, it would still be very good performance. And it is possible in principle. So the argument that humans sometimes disagree doesn't prevent a machine doing well on the task.

Another possible output for this type of task is an opinion or interpretation and a justification of it. A machine might feasibly be considered "good" if it output an unpopular interpretation with a good justification.

Great point! The same type of argument is going in the AI subfield of computational creativity. A computational system might produce a piece (of art or music, say) which nobody initially likes, but could provide a justification/interpretation which might be convincing. In some ways this would be a big improvement over a pure Turing test-style judgement of the output itself. I think Margaret Boden has written this idea into her definition of computational creativity.

EDIT: "The ultimate vindication of AI-creativity would be a program that generated novel ideas which initially perplexed or even repelled us, but which was able to persuade us that they were indeed valuable.", from Boden, Creativity and Artificial Intelligence, 1998. I believe, based on her other writings, that she thinks of it as a sufficient, but not a necessary condition, for AI creativity.

I thought before writing "always". But I do think it will be difficult for the foreseeable future, because as @Grue3 already pointed out, it's ambiguous even to humans.

The central problem of getting a machine to understand metaphor, irony and the like is going to be Hard by any definition.

Being ambiguous to humans doesn't make it an ill-defined task. A machine which perfectly mimicked the judgements of just one human would be doing pretty well, well enough that we couldn't justify saying "judging emotion is a big problem for AI" unless we said the same thing about humans.

Having said that, I agree that it seems a hard problem for the foreseeable future.

> So presumably the purpose of this experiment was to quantify just how bad algorithms can be at inferring meaning?

That's a bit harsh, but I agree.

Words & music are both key to understanding a song's "mood". Only looking at words can completely strip the irony/sarcasm off a song (as can only paying attention to the arrangements and melody/harmony.)

How would such analysis even work on artists like Dylan?

They use Spotify's valence score as a measure of the mood of the music.