Add to this, a very strong insistence that everything is in a perpetual state of flux, that there is no perpetual "you" or "me" (or indeed anything). Trivial physical example - a molecule in your body is exhaled, and now inhaled by me... (cue "we are all one..."). A part of the earth turns into a banana plant, you eat the banana, and the banana turns into a human being. Now where are the boundaries of "you" and "the Earth"? See first paragraph reminding you about the very limited nature of your sense perceptions and memories.
The foregoing is stuff "we all know" but push into the background while we "get on with real life", but the yogic/Buddhist approach asks us to foreground it, and ask ourselves how we would conceive of and live life with those understandings in the foreground. Further, what if dynamics beyond those limited senses were at play? Psychology and ethics which takes the above as a starting point, rather than exisiting power relations, ends up looking different from what we have now.
The whole thing is like a snake swallowing its own tail. If what "they" are saying is correct, then this is "my" dream. The Buddha is a character I dreamed up to remind me how to wake up.
Why isn't this solipsism? First, because it's not my personal dream (the person being a character himself). Second, for a similar reason as MWI is not solipsism. This branch really is my own branch (in a particular sense). What I experience as other people are really "my versions" of them, but this does not mean there aren't other perspectives analogous to mine.
This idealist perspective can be made internally consistent. At that point, deciding between it and materialism may seem like a matter of preference. The difference is that the idealist perspective can be confirmed, in the same way that a nighttime dream can be: you discover, very precisely, how you constructed the whole thing, including all the tricks you used to blind yourself to the ruse.
Egyptians were supposedly obsessed with being lucid during dreams so they don't "lose" themselves in the afterlife. Seems quite logical, since you essentially lose track of yourself during a dream.