I think this has to do with tolerance for being stuck, and that varies depending on how rewarded you think you’ll be for figuring it out and getting unstuck.

Real science and math involves getting stuck on problems, perhaps for weeks, months, or years. I guess we should be happy that there are people who can tolerate being stuck.

My tolerance for getting stuck on a mere game has dropped dramatically since I was a kid; many of the games we played then are unplayable by modern standards. You had to draw maps and take notes, yourself, rather than the computer remembering things for you.

But text adventures back then sometimes weren’t meant to be played alone. The game might be single-player but it was a group activity for college students where you’d share ideas. The modern equivalent might be games where you’re expected to search the web to find recipes and strategies for things.

Yep, we aren't exactly being conditioned towards higher frustration tolerance nowadays... I don't think I could still deal with a PC without an SSD.

I think there are different kinds of being-stuck, though. With many problems in science, there's at least things you can try to gather more data. So you're stuck, but you can come up with new experiments to get new insights into the problem. Here, you're only given the one set of examples and have to make do. I guess you could still see the process of generating hypothesis and testing them against the examples as a sort of "experiment", but it still feels a lot "stuckier" if you don't get anywhere with it.

Don't worry you can always solve these problems trivially: The examples on the left side are all on the left side, while the examples on the right side are not.

Yeah, the author says that for any of these problems "there should only be one reasonable rule." But I suspect that "reasonable" here really points to contingent facts about human psychology, i.e. some rules just strike us as more intuitive or appealing than others, but they aren't correct in any objective sense. That sort of gives the lie to the notion that what we're exercising here is "meta-rationality."

The problem with these puzzles is that, without rules for the system, you can just make up your own rules and then solve the puzzle within the context of those rules.

For example, in the second puzzle, the arrangement of black-and-white shapes is the same on the left and right pages, but the right page is rotated relative to the left page. Is the question about the shapes as in an ordered collection? Or is the question about the pages in their entirety? These problems tend to be underspecified, and end up being more of a guessing exercise about the authors intentions than anything else.

Yeah I think a lot of it is learning to think like how the people who thought up the problem think.

It still may not be a bad exercise (like art students at a gallery copying a master's work), but you shouldn't get too far ahead of yourself claiming it's some sort of 'exercise in pure reason'.

These sorts of tasks are teaching you how to think a specific way which our society promotes. Mechanistic, natural, causal, rational (in the first-order logic sense) with a healthy dose of Ockham's razor and simplicity as an aesthetic.

I stress though, I'm a big fan of these things and the innovations they have enabled, but you still have to understand that they are an axiomatic underpinning. It's like Euclid's parallel postulate in a way: There is non-euclidean geometry out there.

The fact that the solution of these problems is in some sense satisfying also has a lot to do with the fact that the people making these problems are Western systematic-thinkers. They think like us (because we were trained by them).

That's not to say there isn't value in learning this way of thinking, it's gotten society a long way.

How would you characterize the opposite of (or alternative to) "systematic" thinking?

Not an alternative to systematization, but different systems. Just that the analogies or groups that make sense to one group may not make sense to others.

Could you give an example of a different type?

Broadly, for example, https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2838233/

"""

Analytic cognition is characterized by taxonomic and rule-based categorization of objects, a narrow focus in visual attention, dispositional bias in causal attribution, and the use of formal logic in reasoning. In contrast, holistic cognition is characterized by thematic and family-resemblance-based categorization of objects, a focus on contextual information and relationships in visual attention, an emphasis on situational causes in attribution, and dialecticism (Nisbett, Peng, Choi, & Norenzayan, 2001).

"""

This also comes up a lot in cognitive test design.

The anecdote I've always heard in reference to it was

""" What is considered wise in one society may not be considered wise in another; the value and meaning of intelligence depends on cultural norms. Demonstrating the culturally-specific nature of knowledge and intelligence, Cole, Gay, Glick, and Sharp (1971) conducted an experiment in which Western participants and Kpelle participants from Liberia were given an object-sorting task. Participants were asked to sort twenty objects that were divided evenly into the linguistic cat-egories of foods, implements, food containers, and clothing. Westerners tended to sort these objects into the groups for food and implements, while Liberian partici-pants would routinely pair a potato with a knife because, they reasoned, the knife is used to cut the potato. When questioned, Liberian participants justified their pairings by stating that a wise person would group the items in this way. When the researchers asked them to show what an unwise person would do, they did the taxonomic sort that is more familiar to the Western culture. """

quoted from https://uscaseps.org/wp-content/uploads/2020/07/standardized...

That actually explains a lot! Thank you, that's a great anecdote

A simple example that still constrains the puzzle to human abilities (but also makes it less universal) might be this. Rather than diagrams/images, each puzzle consists of two groups of short depictions of two people interacting. The differences are in the relationships, emotional states, or modes of expression. That kind of judgement requires different perception, intuition, knowledge, and so on compared to the puzzles based on shapes. Probably a lot of people who were good with the "standard" Bongard problems would struggle with the "interpersonal" variety, and vice versa.

The flaw with that "solution" is that if you have found the rule, you should be able to say whether a new image belong on the right or the left if it is presented to you. If you say, "I can do that, but I need one more bit of information, namely whether the new image belongs on the right or the left", that's a pretty severe defect.

There's no necessity (in general) that a new image should be able to be classified under the rule. If I give you two finite groups A={1,3,9,-2} and B={7,-11,i,5} and the rule actually is tautological, Then a new number 22 doesn't belong to either group under the rule.

A few of the examples from the article actually are similar. The two circles where one circle is either clockwise or counterclockwise from the nearest indention only admits pictures with two circles, one on the surface of the other and an indentation. There are images which wouldn't fit into either.

A math professor of mine was illustrating this point with number series (of the sort on aptitude tests, eg squares,arithmetic sequences, etc), by listing an obvious sequences whose completion ended up being an obscure function which diverged at the next point.

So, the trivial solution (and the more ultra-complicated solution) is defective basically because it's not interesting under the rules of the game which assumes the answer is somehow interesting, but not impossible to guess.

Thanks for that, actually made me chuckle :)

(And it illustrates the point quite well, as that is indeed probably the simplest and most general rule you can find.)

I can see where you're coming from. I always have to judge my working solutions using the parsimony of features and parsimony of rules to navigate the feature/solution landscape of Bongard problems.

The underlying assumption is that the problem's author has followed the same rules which is an assumption of good faith on my part. This narrows down the feature-solution space considerably or at least biases it in a way where I can prioritize hypotheses in a more tractable way.

Part of what I feel makes me a good puzzle solver is imagining that I'm a puzzle maker. What was going through the author's mind when they conceived the puzzle? If I can start to pull at that thread then the complexity of the puzzle will start to unravel.

Hmmmm yes, this assumption of good faith might actually be an important part of approaching these problems. I think I instinctively look for approaches to problems that would work even for adversarial examples. In the case of these puzzles, that of course doesn't work, because you then imagine the rule to be something completely outlandish and give up before you've even checked the easy options.

Something like "I know it's _possible_ the author has chosen a ridiculous rule, so it doesn't make sense for me to look for it, because even if I find it, I just got lucky and didn't actually solve the problem."

That might be a side-effect of perfectionism, actually.

I was thinking along these lines too when the author went into machines solving the problems. One bit that jumped out was the quote from Hofstadter:

> They depend on a sense of simplicity which is not just limited to earthbound human beings.

Followed immediately by a problem whose solution depends on having a sense of 3-D objects in gravity!

Solving Bongard problems is surely a hard thing to get an AI to do, but I am wondering too about AI-authored instances. Or, say, problems authored by aliens, with a different evolutionary history, and different in-built biases for cognition. Would they necessarily be solvable by humans, or our Bongard problems solvable by them? Some aspects (number maybe?) are probably universal. But even a good-faith puzzle maker has to take some assumption of shared basis for perception.

This probably connects up to the author's final point that "what objects even are" is not absolute.

> Followed immediately by a problem whose solution depends on having a sense of 3-D objects in gravity!

I can think of a Bongard problem where the images on the left are rebuses which spell English words, and images on the right are rebuses which spell Russian words. That's baking a LOT of a priori knowledge into the the puzzle. Alexandre Linhares' A glimpse at the metaphysics of Bongard problems talks a bit about what assumptions can go into this problems:

> An interesting but generally ignored aspect of Bongard problems is that their difficulty for a given subject is directly associated his or hers (or the system’s) previous experience.Since the problems consist of geometric figures, one may be led to believe that cultural factors do not influence the performance of a person attempting to solve them. This is not the case.

> Solving Bongard problems is surely a hard thing to get an AI to do, but I am wondering too about AI-authored instances.

I like where you're going. I left out the search I did of "generative adversarial networks bongard" because no good results popped up. (Hint: this would make for a fantastic HN post if any researcher wants to earn fake internet points).

Finally, this[2] comment stuck with me over the years. As always, would solving or generating Bongard problems be a quantum leap in AI or would they, like so many other problems, be subsumed into the category of AI-solvable problems and we all move on to the next problem at the frontier?

[1] - http://app.ebape.fgv.br/comum/arq/Linhares2.pdf [2] - https://news.ycombinator.com/item?id=8964017

This makes a lot of sense. People have wondered how our cognition would be affected by living in zero gravity.

And our language reflects these cognitive biases; how we would express the experience of "getting high" in zero gravity?

I feel similarly; the author mentions spending ten minutes trying to solve one of these puzzles, and I can’t imagine doing that and enjoying it. Maybe it’s the case that spending more time on the ones that stumped me would yield fruit, but I have the impression that on this class of problem, if I don’t see the solution within two minutes then I probably won’t be able to figure it out in ten, which disincentivizes investing time in them. I’d be interested in knowing if the people who /can/ solve all the problems in the article do so by investing time in them and being methodical or if they just “see it” eventually, which is how I feel solving the easier ones.

There’s also the factor that some Bongard problems, independent of their difficulty factor, are just more satisfying than others. Spoiler for the fourth one, with pairs of circles: its solution is that the entries on the left have $property while the ones on the right...don’t. This makes the right side virtually useless except to check the rule that you derived from the left side.

Maybe it’s just that I don’t have research experience, and am thus unsteeled against problems that seem impenetrable, or maybe I just don’t have the mindset to be good at these, but I agree the really difficult ones can be frustrating.

The cumbersome part is that the first step requires you to pick up patterns and differences without knowing which are actually relevant. Which is fine for the first few puzzles, but gets much harder with increasing amounts of details.

It's basically the same kind of problem that one faces in the more frustrating debugging sessions where you're looking at tons of data and try to find a pattern or clue of what causes the bug under what circumstances.

At least with Bongard problems you don’t have heisenbugs.

You're certainly not the only one, but the point of the article is that solving these kinds of problems, with all the vagueness that implies, is an important feature of human intelligence.

Since I generally seem to do fairly well in problem solving, that is exactly what I'm trying to figure out: Are these problems actually representative of an important skill that you need for general problem solving, or are they, through their nature of being man-made puzzles, actually in a realm of their own?

When I'm looking at some pattern that I'm trying to find a rule for in real-life, I don't think I'm running into the same frustration and in fact greatly enjoy trying to figure out rules for how things work (or so I believe, at least).

I think a crucial difference is that I know that the problems I encounter in real-life are only "as complex as necessary", and the data I'm looking at is a direct result of some process that serves a specific goal; presumably one I think "makes sense", as I wouldn't look for a rule otherwise. In contrast, puzzles are made to be complicated on purpose, and I suspect that annoys me subconsciously to the point where my brain complains about engaging with it. But it's only these kinds of "figure out the rules" puzzles, so there has to be another important difference compared to logic puzzles. Possibly the difference is: for the logic puzzle, the "meta-rules" for the problem are made explicit and I know the solution-space exactly. For the Bongard problems here I found myself thinking for example: "wait, is it always just two groups distinguished by single rule, or can there be dependencies on the positions of the symbols within the groups as well? What kind of solution am I even looking for?", and that also apparently frustrates me.

Sorry for the wall of text, but I've actually been trying to figure out why these kinds of problems get on my nerves for quite a long time, lol.

Bongard puzzles are pretty much the same as the test matrices in IQ tests, which annoy me in the exact same way these Bongard puzzles do. If you'd ask people questions in the same manner these puzzles do, they would refuse to answer because they'd feel trolled. Arguably, that's the case.

They're closely related to Raven's Progressive Matrices https://en.wikipedia.org/wiki/Raven%27s_Progressive_Matrices which are indeed the source for IQ tests. But the point of this all is that rationality works within a frame; Bongard games are just an illustration that the real problem of meaning and choice and acting in the world is the not-rational one of choosing a framing.

I don't know, the first took me two seconds, the second five and the third five too. I stopped reading there because the article says next ones are harder and I would like to take a quiet time to read the whole thing.

I use to solve chess problems at lichess, a similar concept. Maybe.

Is it really an infinite haystack? It's context. And simplest solutions first, gradually think of more complicated ones. First try to find a pattern visually, then use simple concepts, then more complex concepts.