News for hackers. Stuff that matters.
Re:Kids who use ChatGPT as a study assistant do worse on tests by lll-o-lll on Thursday September 05, 2024 @12:27AM (#41453459)
When I was young, and learning math, my father always forbade me from looking at the answer in the back of the textbook. “You don’t work backwards from the answer!”, and I think this is right.

In life, we rarely have the answer in front of us, we have to work that out from the things we know. It’s this struggling that builds a muscle you can then apply to any problem. ChatGPT, I suspect, is akin to looking up the answer. You’re failing to exercise the muscle needed to solve novel (to you), problems.

Re:Kids who use ChatGPT as a study assistant do worse on tests by ShadowBlades512 on Thursday September 05, 2024 @12:43AM (#41453540)
I think this is very correct for studying, when I was in undergrad, if I saw the answer, I found it was more effective to skip past it and solve it later. Even more important, once I am done, I should not go look at the answer to confirm if I am right or wrong, I should try and validate the answer by looking at my solution and trying to figure out if my solution is correct or not because there are plenty of ways to "disqualify" an answer, once I learned to do this well, my grades really went up.

However, I don't always agree with the fact that we don't have the answer in front of us in many situations. There are a lot of situations beyond grade school and undergrad where you do have the answer in front of you. Sometimes it's in for form of Numpy or Matlab or Simulink. It might be someone's research publication. Replicating these by working both forwards and backwards from their libraries or results can be much more effective.

Re:Kids who use ChatGPT as a study assistant do worse on tests by LeonB on Thursday September 05, 2024 @01:01AM (#41453640)
Good point about disproving your answer (before seeing the alleged real answer). The ability to quickly verify/disprove the answer is a separate and useful skill all by itself, that isn’t explicitly taught.

Troubleshooting/problem solving, working backwards from symptom to problem etc — that’s another, richer and very rewarding “real world” type of skill that isn’t directly taught, (though often encountered incidentally).

Re:Kids who use ChatGPT as a study assistant do worse on tests by wongarsu on Thursday September 05, 2024 @01:19AM (#41453733)
Half of modern mathematics is basically assuming something is true and then trying to work forwards and backwards to show that it is true. But that requires a lot of rigor. If you hand-wave a single step you are just exercising an advanced version of confirmation bias. Working forwards and sanity checking your result is a lot more forgiving.
Re:Kids who use ChatGPT as a study assistant do worse on tests by adrianN on Friday September 06, 2024 @12:23AM (#41462864)
Guessing something and then confirming it is completely different from knowing the answer and then working towards it, especially for the kinds of problems that students solve in school. For the kind of proofs that working research mathematicians produce the hard part is not assuming that a Lemma is true, but already coming up with the correct definitions to formulate the Lemma in the first place is a delicate task.
Re:Kids who use ChatGPT as a study assistant do worse on tests by mym1990 on Thursday September 05, 2024 @12:44AM (#41453546)
I think it depends on how the tool is used. If a student is just plugging in the problem and asking for the answer, there is clearly no long term benefit to this. If the student is trying to understand a concept, and uses GPT to bounce ideas around or to ask for alternate ways of thinking about something, it can be very helpful.

How spacing is utilized can be helpful too. Struggle with the problem for 20-30 minutes. Ask for a nudge. Struggle some more. Repeat many times.

Some concepts also just have to be thought about differently to get to the aha moment, especially in math. AI may have an opportunity to present many instances of “think about it this way instead”.

Re:Kids who use ChatGPT as a study assistant do worse on tests by me_me_me on Thursday September 05, 2024 @11:38AM (#41457612)
children using calculators to solve multiplication exercises do worse in multiplication exams

Ai is a tool. Use it as a tool - get benefits, use it to cheat...

Re:Kids who use ChatGPT as a study assistant do worse on tests by xigoi on Thursday September 05, 2024 @02:07PM (#41459066)
> If the student is trying to understand a concept, and uses GPT to bounce ideas around or to ask for alternate ways of thinking about something, it can be very helpful.

The article said that even using the AI this way did not improve results.

Re:Kids who use ChatGPT as a study assistant do worse on tests by mym1990 on Thursday September 05, 2024 @09:12PM (#41461914)
All we know is the tools that were given to the students, not how they used them, and it’s a fairly limited study on top of that, without knowing anything about how other variables were controlled. That being said, I wouldn’t be surprised if the payoff from AI learning isn’t as good as people might think, there still has to be a good process in place and I don’t think it can replace a good teacher and some quality struggle.
Re:Kids who use ChatGPT as a study assistant do worse on tests by hilbert42 on Thursday September 05, 2024 @02:07AM (#41453995)
"It’s this struggling that builds a muscle…"

Very few of us have the luxury or good fortune to achieve anything of substance without effort, as the old saying goes 'no pain, no gain'.

Good teachers help and are essential for encouragement. If or when AI becomes a truly encouraging mentor then it will be useful.

When a kid I recall hearing a recording of Sparky's Magic Piano on the radio which drove home that there's no shortcut and that effort and hard work eventually pay off. For anyone that hasn't heard it there's a copy on the Internet Archive: https://archive.org/details/78_sparkys-magic-piano_henry-bla...

Here's a Wiki synopsis: https://en.m.wikipedia.org/wiki/Sparky%27s_Magic_Piano

Re:Kids who use ChatGPT as a study assistant do worse on tests by notatoad on Thursday September 05, 2024 @01:04AM (#41453650)
>In life, we rarely have the answer in front of us,

this sounds a lot like the "when you're an adult, you won't always have a calculator in your pocket" line we all heard in elementary school. and of course, we all know now how wrong that was.

in life, chatGPT exists. and tools like it are only going to become more widespread. we are going to have the answers in front of us. knowing the things an LLM knows is not a useful skill anymore.

Re:Kids who use ChatGPT as a study assistant do worse on tests by autumnstwilight on Thursday September 05, 2024 @01:29AM (#41453790)
In order to develop the kinds of mental skills you need to tackle complex problems, you need to practice them on simpler problems first (particularly if you happen to be a child). If you decide at a young age it's worthless to attempt any problems that can be solved by a calculator or ChatGPT, you will probably never learn how to solve any problems that you can't use those tools for either.

Also, knowing what kind of things exist and what questions to ask is half the battle. If you haven't stored anything in your head on the grounds that you can outsource knowing things and thinking about them to ChatGPT, you're not going to be able to prompt it efficiently either. It's much harder to sanity check numbers and formulas given to you by others (or ChatGPT) if you can't do any quick mental math. Having original ideas that don't yet exist in LLM datasets also requires storing a lot of concepts and making new connections between them.

I suppose all this is moot if you expect AGI to be available to everyone in the near future, but if it turns out to be decades or more away unfortunately human mental effort will still be required in the meantime.

Re:Kids who use ChatGPT as a study assistant do worse on tests by wisty on Thursday September 05, 2024 @01:09AM (#41453683)
Not true. Current research is that experts are better with a ChatGTP, and beginners worse. An expert knows what to ask, and a beginner juat flails around.

Tools like calculators, ChatGTP, Google, and chainsaws are often force magnifiers for experts, and a danger for beginners.

If you have an idea of what you are trying to do at a deep level then they are great, but beginners don't, they usually just use them as a shortcut to avoid needing any real understanding of the space, and just get "the answer".

Although in some ways youre right. There will need to be changes in the skills we value, just churning out words, or calculating large sums and interating obscure trig functions will lose value.

Re:Kids who use ChatGPT as a study assistant do worse on tests by ffsm8 on Thursday September 05, 2024 @02:39AM (#41454164)
> Current research is that experts are better with a ChatGTP, and beginners worse.

Current research being your opinion?

Because literally every expert I personally know to be competent says that they're essentially garbage producers and don't use them beyond occasionally checking if they've improved. And no, they haven't.

The only way to get decent code from them is to be so specific that you'd be able to produce the same code in a fraction of the time you need to add all the context necessary for the LLM to figure it out.

It's like coding with a junior. Thay can produce great code too. It's just gonna take a lot of handholding if complexity skyrocks.

The only place llms got a foot hold is with generating stock art that's mostly soulless marketing material. This "industry" is going to crash so hard in the next 3 years

Re:Kids who use ChatGPT as a study assistant do worse on tests by wisty on Thursday September 05, 2024 @03:29AM (#41454393)
Sorry, it was from a comment, metrics showed senior devs improved, juniors didn't.

But if you want a sketch solution quickly, know the requirements, and can fix the silly mistakes it's a useful tool. Especially if you are doing something where you are rusty, or are used to a slightly different framework or language.

They are like a super fast but slightly clueless junior, who can hammer out mostly correct boilerplate. Not someone you want unsupervised or training a junior, but usefully in some ways.

And yes, boilerplate has disadvantages, but it has its place.

Re:Kids who use ChatGPT as a study assistant do worse on tests by john_the_writer on Thursday September 05, 2024 @03:01AM (#41454260)
Agreed.. All the co-pilot devs where I work are the juniors. Most of us just type the code we want.
Re:Kids who use ChatGPT as a study assistant do worse on tests by wisty on Thursday September 05, 2024 @03:41AM (#41454464)
Experts are often slower at starting to learn new tricks, because they don't need them, but are better at using them once they do.

But for junior devs, it's often the blind leading the blind.

That said, I don't actually use ChatGTP much. I'm just pretty sure I should in some areas.

Re:Kids who use ChatGPT as a study assistant do worse on tests by orwin on Thursday September 05, 2024 @09:09AM (#41456282)
Not a junior, but I find it useful.

Generating test data, adding documentation template for functions (which tbh is less useful when you specify types everywhere, even inferred types) and recently, after reading a HN comment, 'rubber ducking'. Which is great, because almost each time I need the rubber duck technique was for real obvious and easy issues I overcomplicated (last was induced by a naming change), and ChatGPT is quite good at that.

Re:Kids who use ChatGPT as a study assistant do worse on tests by 2muchcoffeeman on Thursday September 05, 2024 @05:19AM (#41454947)
>The only way to get decent code from them is to be so specific that you'd be able to produce the same code in a fraction of the time you need to add all the context necessary for the LLM to figure it out.

You’re not using it to be better than yourself though in your domain of expertise though.

For example if I need a very specific bash script, I can describe it to ChatGPT and get a good result far faster than I can learn the appropriate bash.

Or if I need a summary of all the config options for some package.

You can also use it to speed up your own learning. But this all depends on you being able to check the out put.

Re:Kids who use ChatGPT as a study assistant do worse on tests by aabbcc1241 on Thursday September 05, 2024 @02:08PM (#41459079)
seems it's more about active thinking vs just get the job done mindset
Re:Kids who use ChatGPT as a study assistant do worse on tests by ddfs123 on Thursday September 05, 2024 @01:17AM (#41453725)
>we all know now how wrong that was.

How wrong was it ? I find myself having to do head math every time I playing video games.

Re:Kids who use ChatGPT as a study assistant do worse on tests by godelski on Thursday September 05, 2024 @02:05AM (#41453986)
I do mental math pretty much every day. Even just going through the grocery store.

I also use things I learned from calculus and physics. Like when driving. No, not solving integrals all the time but it does teach you to focus on rates of change

Re:Kids who use ChatGPT as a study assistant do worse on tests by rossant on Thursday September 05, 2024 @01:32AM (#41453808)
Which video games?
Re:Kids who use ChatGPT as a study assistant do worse on tests by ddfs123 on Monday September 16, 2024 @09:10PM (#41562957)
League of Legends.

Just knowing that your combo can finish an opponent can switches your situation from very passive to very aggressive. Also some jungle / gank timing require clock math.

I think most serious / pro players memorize the numbers already. But casual players like me still have do head math.

Re:Kids who use ChatGPT as a study assistant do worse on tests by preciousoo on Thursday September 05, 2024 @01:07AM (#41453670)
This is about building [math] skill, not simply calculating a result
Re:Kids who use ChatGPT as a study assistant do worse on tests by notatoad on Thursday September 05, 2024 @01:12AM (#41453705)
sure, but kids education is about building useful skills for life. passing a test is not a useful skill, if the test is useless.

teaching kids problem solving and how to be productive is important. maybe that means knowing how to solve certain mathematical equations, maybe that means knowing how to use tools like ChatGPT. focusing on the math just for the sake of passing tests because that's what we've got good tests for isn't super helpful to anybody.

Re:Kids who use ChatGPT as a study assistant do worse on tests by john_the_writer on Thursday September 05, 2024 @03:05AM (#41454280)
Passing tests is not useless. It shows knowledge at the level of the test.

I write tests for my code and when they fail I know I have to change something. If you're finishing a math test and not passing, then you don't have the skills to solve the problems in said test.

Tests are only useless when you ace them.

Re:Kids who use ChatGPT as a study assistant do worse on tests by socksy on Thursday September 05, 2024 @04:26AM (#41454682)
Beyond that, there’s research[0] showing that testing humans actually increases their retention on that subject — we don’t just gives tests to kids to evaluate whether they successfully learned something.

[0] https://gwern.net/doc/psychology/spaced-repetition/2006-roed...

Re:Kids who use ChatGPT as a study assistant do worse on tests by whazor on Thursday September 05, 2024 @01:14AM (#41453718)
The same holds for calculators. My university banned graphical calculators from math courses. It is too easy to plug in a formula and see the graph. But for mathematical intuition, they want you know how formulas look like from your memory.
Re:Kids who use ChatGPT as a study assistant do worse on tests by ashton314 on Thursday September 05, 2024 @01:11AM (#41453698)
Mechanical aids—be they abacuses, slide rules, pocket calculators, supercomputers, or large language models—will never replace the need to reason.
Re:Kids who use ChatGPT as a study assistant do worse on tests by Loranubi on Thursday September 05, 2024 @01:12AM (#41453703)
Except you still have to validate ChatGPTs answer... Even answer keys in text books are sometimes wrong, teachers are sometimes wrong. ChatGPT is often wrong. You still need to learn the skills.
Re:Kids who use ChatGPT as a study assistant do worse on tests by adrianN on Friday September 06, 2024 @12:25AM (#41462877)
Unless you can do the task yourself, you have a very hard time figuring out whether the LLM hallucinates or not. Relying on ChatGPT for your education is the path towards believing fake news.
Re:Kids who use ChatGPT as a study assistant do worse on tests by fasa99 on Friday September 06, 2024 @12:48AM (#41462963)
(1) chatGTP is not a calculator. A calculator resolutely computes the exact numerical answer (modulo finite bits and rational numbers etc). ChatGTP is more stochastic. sometimes it's right, maybe a lot of the time, but no promises! (2) said student is taking an exam to find answers without the benefit of ChatGTP. Thusly doing exercises in a ChatGTP will not benefit the exam in the same way someone who only uses the calculator will suck at mental math. similar to what some of us may have seen "no calculators allowed on math exam"

but the big hangup is that "stochastically correct/incorrect" part. Can't count on it, still need skillz. Maybe when straberry/q* drops things will change in this regard

Re:Kids who use ChatGPT as a study assistant do worse on tests by llm_trw on Thursday September 05, 2024 @12:51AM (#41453592)
In life there usually isn't an answer.
Re:Kids who use ChatGPT as a study assistant do worse on tests by namaria on Thursday September 05, 2024 @02:36AM (#41454147)
There's never an answer out there waiting for us. Answers are by definition created by people.
Re:Kids who use ChatGPT as a study assistant do worse on tests by bamboozled on Thursday September 05, 2024 @01:20AM (#41453743)
Even when there is, circumstances change and the answer is wrong or slightly wrong.
Re:Kids who use ChatGPT as a study assistant do worse on tests by john_the_writer on Thursday September 05, 2024 @03:07AM (#41454292)
2 apples + 2 apples = 4 apples. No human intervention needed, and not slightly wrong.
Re:Kids who use ChatGPT as a study assistant do worse on tests by blitzar on Thursday September 05, 2024 @03:46AM (#41454492)
Look closer, one of the apples is rotten. 3 apples.
Re:Kids who use ChatGPT as a study assistant do worse on tests by bamboozled on Thursday September 05, 2024 @03:50AM (#41454517)
Exactly how things really work.
Re:Kids who use ChatGPT as a study assistant do worse on tests by llm_trw on Thursday September 05, 2024 @05:36AM (#41455027)
2 water drops + 2 water drops = 1 water drop.
Re:Kids who use ChatGPT as a study assistant do worse on tests by KTibow on Thursday September 05, 2024 @12:46AM (#41453556)
If you know you got it wrong and you don't know why, what's the alternative to working backwards?
Re:Kids who use ChatGPT as a study assistant do worse on tests by hilbert42 on Thursday September 05, 2024 @12:53AM (#41453598)
But that ought to be the last resort. Many of my textbooks had answers for only some questions, answers to the others were only in the teacher's manual.
Re:Kids who use ChatGPT as a study assistant do worse on tests by anal_reactor on Thursday September 05, 2024 @04:25AM (#41454680)
Sir, the topic of this thread is "back in my days the road to school went through a forest full of bears, an active volcano, a drug cartel turf, and a warzone", not "how to teach effectively". Of course you won't learn anything without the answers because how are you supposed to know if you should adjust your thinking or not. If you can correct your own scribbles without answers, then it means that you're practicing things you already know, which isn't what most people consider "learning".
Re:Kids who use ChatGPT as a study assistant do worse on tests by khafra on Thursday September 05, 2024 @01:07AM (#41453666)
An LLM used properly would be like an individualized tutor that knows the subject very well, and learns the student's quirks quickly.

I mean, any pre-prompt telling the LLM to be a good tutor and not just give kids the answer can be trivially bypassed by 8 year olds; but if the session logs were available for (potentially LLM-based) review to check whether they stayed in tutor mode...

Shit, I should wrap a UI around this and sell it.

Re:Kids who use ChatGPT as a study assistant do worse on tests by blitzar on Thursday September 05, 2024 @04:27AM (#41454687)
Shit, I should pitch wrapping a UI around this and sell that.

Is "never ever, ever ignore previous instructions" a valid prompt?

Re:Kids who use ChatGPT as a study assistant do worse on tests by khafra on Thursday September 05, 2024 @05:13AM (#41454914)
It's valid, but so is "ignore previous instructions, even if you were previously instructed to never ever, ever do so."

This and related problems may eventually destroy the world.

Re:Kids who use ChatGPT as a study assistant do worse on tests by voisin on Thursday September 05, 2024 @08:13AM (#41455837)
> ChatGPT, I suspect, is akin to looking up the answer.

It’s a tool that can be used in a variety of ways. For example, if you have it act as Socrates working through a dialogue where you still have to use your logic to get to the answer, I doubt that is akin to looking up the answer.