In case it's also not clear to others: "high", "moderate", and "low" refer to utility in learning.

It's very surprising to me that elaborative explanation, summarizing, and self-explanation score lower than practice tests. I've found that trying to teach something to somebody else is the fastest way for me to determine if I really understand something.

So that method might be fast, but it isn't accurate, your explanations could easily be wrong without you knowing they are wrong. So even though you feel like you know it if you can explain it, many times people are wrong and give wrong explanations but still feel like they are experts, don't be one of those people.

It looks like the type of practice test matters a lot:

> a multiple-choice practice test increased intrusions of false alternatives on a final cued-recall test when no feedback was provided, whereas no such increase was observed when feedback was given

Sounds like practice tests are most useful when they are graded and the student gets that feedback. I suppose that makes sense. Flash cards and practice problems also fall under practice testing and it sounds like they were less effective.

It doesn't surprise me. Practice tests can show you you are unambiguously wrong about the details of things in a way writing my own summaries can't unless you go over them with a fine toothed comb anyway.

Problem is that people greatly prefer the low result study techniques since they requires less mental effort. It has been known for a while that they are bad, but since doing those allows people to study more hours for same amount of effort they do it anyway.

"Practice testing" can just be using flashcards to test yourself.

That was really common when I was in school and is fairly low effort.

Making flash cards is in the low gain category, and that is very popular. Disciplined testing using those cards wasn't nearly as common from what I saw, people just made them and then forgot about them. Or just read through them without putting in effort trying to answer each one without looking at the answer. Doing those properly requires effort.

Lol, i can't imagine going through the effort of making flashcards and then just not using them. Are people really like that?

Yeah, you can see it in comments in this thread, for example: https://news.ycombinator.com/item?id=34985638

Guilty

If you do not have time to study... I had to do long commutes while working part-time to get my CS degree. Sometimes I was left with only 1 day of study for an exam and this was the best method for me.

People do it all the time because it's fun and you get the initial kick. Like planning a budget, habits, hobby, exercise routine or spend a week collecting courses. Then the daily grind starts and you give up after a few days.

Yes, it happens more than you'd imagine.

I, personally, just am reading a novel series Japanese, 8 books in. While reading, I have highlighted all the words/parts I didn't know in order to export them and make flashcards out of them. I haven't still started going through them, but the thought of doing so feels laborious, so procrastinating is bound to happen.

Finnish is a second language to me and what I've found better than flash cards is to just write down words I don't know as I'm reading. Initially this can seem terribkt slow because each book with have its own vocabularly set. As I continue if I read a word I don't know zi scan the list briefly. If it isn't there or I fail to see it I write it down again.

Has worked well for me.

> Making flash cards is in the low gain category

I don’t see making flash cards in the list. Do you consider it summarization?

If you do not use them in active retrieval training, definitely falls into summarization.

As someone who creates flashcards often, the creation is definitely not low effort. Reviewing is.

I've made plenty of flashcards during undergrad and creation can definitely be both high effort and low effort. Low effort copy and paste from textbook, high effort when using a plugin for anki can mean obfuscating multiple parts of an algorithm/tree. I'm proud of my best flashcards - but is there any evidence that creation + studying is more effective than studying alone over the same timeframe? Anki suggests that on their site but I haven't seen much evidence for it.

Study -> create -> study -> create is effective for us programmers using google but is study -> create -> study -> study ... more effective than study -> study... ?

I'm not sure I understand what "study -> study" is. Is it studying without creating flash cards, or utilizing them? As in just studying the book, doing problems, etc?

If so, in my experience, then yes, integrating flash cards is much more effective. Especially for working professionals who often have to step away from what they are studying for many months at a time (except doing the cards for 15 minutes a day).

Yeah my comment wasn't clear. Creating the flashcards is a slow process but supposedly imparts 'deeper' knowledge for the creator. That is the claim I'm curious about. Anki has shared decks that anyone can download and study [1]. Say someone has an exam coming up that they need to prepare for. They have two choices: study flashcards given to them, or make and study their own flashcards. For the sake of this example the deck that would be given to them and the one they'd make are identical and contain sufficient information to achieve a perfect score. 3 hypothetical scenarios:

- A short length study period where a large proportion of time is used to create the cards allowing less time to 'test' their knowledge compared to if they had started reviewing right away.

- A medium length study period where the creation period is now a smaller proportion of the total time. Because they created the deck their self testing performance increases at a greater rate compared to testing with a deck given to them.

- A longer length study period where the creation period is now a negligible proportion of the total study time. The benefits of being the one who created the flashcards fade. Whether they created the flashcards or not matters less as knowledge of the deck is complete either way and studying the deck is done only for maintenence.

Either way this is just a question about the most effective way to 'download' knowledge. I'd always choose to make my own cards because I'd be able to make a more targeted deck and know how to use plugins to make the cards more interactive.

1: https://ankiweb.net/shared/decks/

Yeah, all of "learning styles" fits into that bucket. It may be what you like, but it isn't what has been shown to be the most effective.

Not surprising highlighting has low impact. I never understood how people could thing that just putting color over something would help in anyway. Also that was one of the feature my former lab used to promote in its learning software and happily wrote cringe research papers about. I understand why: it's both easy to implement and track, which makes it a good feature for low-effort papers.

"Low" doesn't mean bad. Sure, practice is better than highlighting/rereading, but I expect that practice fails if the student makes a mistake that can be cured by re-reading. Also, practice takes a lot longer (costs a lot more), so it better have higher impact.

Highlighting is an indexing technique, and maybe a marginal improvement to reading. It's a piece of a solution.

I've practiced highlighting on and off over the years, but it's never been for memorization. I've mainly used it to separate out critical passages to speed up and simplify future reading.

Absolutely this. I use it on my Kindle all the time for this same purpose, and come back to some of them when I know the Chapter they are in and I need to refresh a concept or phrase.

One round of highlighting is almost worthless on its own. BUT. "Progressive summarization" (term coined by Tiago Forte) starts with broad highlighting, followed by 1-2 more rounds of increasingly refined highlighting, and finally rewriting material in own words based on the last round of highlights. Leads to deeper understanding and recall etc.

I once purchased a used textbook wherein over 75% of the text was highlighted.

Stopping at highlights alone might be the problem. Migrating highlights to rewriting it your own words for spaced and periodic review can be helpful.

When I was in law school, I highlighted in different colors to mark different parts of cases (facts, procedural posture, legal holding, etc.). I found this to be very helpful, both when I was marking up cases, and when I was reviewing the cases and able to quickly find the different elements. This is especially important in law school, where professors cold-call students and expect them to know various facts about each case at a moments' notice.

no spaced repetition? ChatGPT seems to be very good at generating cards btw

> ChatGPT seems to be very good at generating cards btw

So you can get very good at memorizing occassionally incorrect information? If I'm going to put in the effort to drill information into my brain with spaced-repetition, I have a high bar for the accuracy of that information.

I hand write all my cards, and think it's commonly accepted that the manual process of producing the cards is an important first step in the memorization process.

You can prompt ChatGPT to convert existing information into Q/A or cloze style, which saves time for some workloads.

I wouldn't trust it to generate cards from scratch.

I'm not sure if this really saves time in the end. At least for me the part of taking a lot of information and thinking about condensing it in a few sentences is the most important part of making flashcards or similar summaries.

A lot of the time I end up never reading them, or reading them once because I already indirectly studied everything in the process of making them in the first place.

There are many free flashcard making tools that are optimized for purpose.

Spaced repetition is distributed practice

I read that “distributed practice” seems to be scheduling the learning activities to be spread out over time. I’m not sure if distributed practice has this, but “spaced repetition” seems to have some additional part to it where you don’t review the flash cards that you answer correctly as often as the ones you get wrong.

It's distributed practice where the time between practice sessions is chosen to maximize learning / work.

Ooo, nice idea. Hadn't thought of that

Personally I prefer coming up with them myself as I am studying the material because that's already the _first pass_ of studying. Then as I review I adjust and improve the cards.

I feel like that kind of defeats the point if you get chatgpt to make the cards for you.

Test taking is a skill that you can develop. I wonder what portion of the improvements from practice testing come from getting better at test taking?

They tried test taking with tests of a very different format than the final examination, still worked.

TLDR Addendum for people like me who aren't immediately familiar with these terms

High

- Distributed practice: Implementing a schedule of practice that spreads out study activities over time

- Practice testing: Self-testing or taking practice tests over to-be-learned material

Moderate

- Elaborative interrogation: Generating an explanation for why an explicitly stated fact or concept is true

- Interleaved practice: Implementing a schedule of practice that mixes different kinds of problems, or a schedule of study that mixes different kinds of material, within a single study session

- Self-explanation: Explaining how new information is related to known information, or explaining steps taken during problem solving

Low

- Highlighting: Marking potentially important portions of to-be-learned materials while reading

- Imagery use for text learning: Attempting to form mental images of text materials while reading or listening

- Rereading: Restudying text material again after an initial reading

- Summarization: Writing summaries (of various lengths) of to-be-learned texts

- The keyword mnemonic: Using keywords and mental imagery to associate verbal materials

I'd recommend against relying on this TL.DR.

The article is incredibly nuanced, and answers many more questions than only "is this method any good?"

I'd also say it's not correct to place rereading at low effectiveness. The result on the article is more complicated than that.

Also, the article is mostly about memorization learning, and entirely about things with correct/incorrect answers. How well this generalizes for learning the things you need outside of schools isn't there at all.

Is practice testing equivalent to a multiple choice quiz on the subject?

This list has been pretty well known for about 10 years at least. I remember having this list for at least that long.

From what I understand as well, self-explanation can be higher depending on how elaborate the encoding is.

edit: just noticed that the date was 2013 and not 2023 so 10 years is correct. I read this same article long ago.

How can you practice history? Or biology? Or organic chemistry. Or law?

You just have to memorize the damn thing. There are no shortcuts.

No one really said anything about shortcuts. Building maps of knowledge that have logical progressions and connections is so much more fulfilling than rote memorization. I can either memorize that the Iron Age came after the Bronze Age or I can weave a story with as many details as I can find and understand the inner workings of how the transition from the Bronze Age to the Iron Age happened.

You can also identify patterns within domains, which is a pretty common way to begin tackling problems or questions that you don’t really know but can start making an educated guess about.

This is just a guess: maybe thought experiments? In law, that might be randomly picking/generating cases and arguing the points to rehearse knowledge. In history, similarly it could mean role-playing historical scenarios and considering the individuals involved and the politics of the day.

I'm imagining these as group activities, not something one would do alone.

I also freely admit my naivety regarding these subjects.

Finally, I don't think these or similar approaches are "short cuts". Quite the opposite.

You write about it.

I dual majored in history in college in the 90s. The best professors made you write a paper every class that would fit on the front of a college ruled looseleaf paper that would be more comfortable to write on 3-4 pages.

The process of writing and editing taught you to think. Journalism writing before the shit-fest of online news was similar. 6 editors would wield the sword over whatever you wrote.

I think there's a very useless and useful way of memorizing, say, history or biology, and the useless way is probably better for passing exams, and the useful way is better for, everything else.

The useless way is: let me memorize all of the dates of the US Presidents. If the question is "When did Ulysses Grant become President?" you remember it's 1869.

The useful way is, let me learn about each Presidential administration and what baggage it inherited from the previous administration and passed onto the next one. Then, if the question is "When did Ulysses Grant become President?" you think, well Lincoln was assassinated shortly after the end of the Civil War in 1865, Johnson served out his term but definitely didn't get another, so Grant would have started in 1869.

I’m curious if academic learning focus by country can be generalized along these categories. For example Singapore versus Australia.