Perhaps try using ChatGPT to identify whether others can do this and for some ideas for why they may not be able to.
I have a bachelors in physics but I wasn't a great maths student in university (the folly of youth) so my linear algebra could be better. On the other hand, I'm not going to redo linear algebra as a pre-requisite for this course. I also haven't done the pre-requisite QC courses, though I'm much more comfortable in that domain.
I don't use ChatGPT to teach me the concepts in the course, I use it as an empathetic tutor to fill in the blanks. If I see a linear algebra identity I'm not familiar with I can ask ChatGPT, telling it the context and what I already know and it'll give me an answer in that context with solid mathematical grounding but (because I ask for it) intuitive justifications. The alternative of stackoverflow, wikipedia, or other online notes can be hard to search for and even when I find a good answer, it's often needlessly mathy and complex. I don't have to worry about hallucination because it is just a gap and once I understand it, usually what the lecturer said makes perfect sense.
In one lecture the lecturer had a throw-away line about stabilisers corresponding to syndrome measurements but didn't elaborate much further. I didn't get how the corresponding circuit would be constructed so I went to ChatGPT and asked it my question with what I knew and the context. In that case, it pushed back and told me that what I knew was wrong. I had formed a misconception which it corrected and gave me an example to show why I was wrong.
I guess I don't use it as a lecturer, or even a tutor. I use it as the really smart guy you sit next to in a lecture theatre who you whisper questions to.
After about an hour of delving into details, examples, history and analogies, I get it on a deep level that would have normally taken many days to develop. Most of the material I found before took an approach of “just memorize this” while I was able to get more parsimonious core concepts that were hidden behind 500 years of practice.
When I stopped thinking about an asset as "a truck" and started thinking of it as "an explanation that cash is missing because we bought a truck" it all clicked.
Before that I thought of assets and revenues as good, and liabilities and expenses as bad. I couldn't make sense of why an asset turns into a liability instead of into a revenue. It's similar to learning intro physics and trying to understand how anything moves if every force generates an opposite force.
As you start to grok a topic, you should start to recognize internal inconsistencies and then you can probe those details. When you feel you have some understanding, you can then start consuming media on the topic at a higher level and any problems in your understanding should fall out fairly quickly.
Just for fun, I asked ChatGPT about it. This is its answer:
> According to the USB 2.0 specification, a remote wakeup event can be initiated by a USB device in a suspended state to request the host to resume communication with the device. The device sends a remote wakeup signal to its hub, which then forwards the signal to the host.
> However, if the upstream port of the hub is in the process of being suspended, the remote wakeup signal from the downstream device may not be received by the host. This is because the hub's upstream port is no longer able to forward the signal to the host during the suspension process.
> As a result, the remote wakeup event may fail, and the downstream device may remain in a suspended state indefinitely. To avoid this race condition, the USB 2.0 specification recommends that downstream devices wait for a certain period before sending a remote wakeup signal to their hubs to ensure that the hub's upstream port is not being suspended.
It sounds as if it made perfect sense, except to know that it actually completely doesn't you have to spend hours reading the spec first.
We're going to see a huge spike of confidently wrong "experts" if they learn by asking ChatGPT to explain things to them.
This has been my experience as well. As I work on some projects I’ve been asking it basic questions for things I’ve already learned or know quite well. Once you deviate past the basics (content you’d find in common tutorials) the hallucination rate is out of control.
The surreal part is that it all sounds so confident and plausible. The way it puts words together satisfies my “that sounds about right” reflex, but the actual content is incorrect or illogical quite frequently.
If I continue prodding I can get it to come up the right answer many times, but I have to ask it a lot of leading questions to get there.
And most is probably not even done on purpose.
That's inherent in the approach. Large language models reflect the majority opinion of the training set. If there's not enough source material in an area that the same thing hasn't been covered in a few different ways, the thing gets lost.
This is well known. What's being done about it?
With prompting maybe we could tease out the minority views, but I could also see it just confidently making shit up to a higher degree.
It took me a while to realize that what I was experiencing was this particular problem, but I already did all the hard work there and only asked it to explain how it fails.
I also recently tried to use it to write a code for drawing wrapped text formatted a'la HTML (just paragraphs, bold and italics) in C, again, just to see how it does. It took me about 2 hours to make it output something that could be easily fixed to work (it didn't without edits, of course) - and only because I already knew how to tackle such task. I can't imagine it being helpful for someone who doesn't know how to do it. It can help you come up with some mindless boilerplate faster (which is something I used it for too - it did well when asked to "write a code that sends these bytes to this I2C device and then reads this address back"), but that's about it.
It's a machine that saves you multiple copy'n'pastes, so you can just do a single copy'n'paste.
Now, overall this is an improvement - since the old way would have been for an amatuer to do a quick google search, and come up with a false conclusion, or get no understanding at all.
That's what I'm really worried about. We know very well what our system can do and be trusted to do, but if you know just enough to ask the wrong questions, you'll get your head twisted up.
ChatGPT gave me a very plausible-sounding result, but mixed up the meaning of two very important values of key parameters which are unfortunately not very well named, so the mix-up can go unnoticed very easily. I did only notice it when some things didn't quite match up and I decided to carefully compare all the result from ChatGPT with the actual spec content. After all, ChatGPT didn't save me any time, it rather cost me quite some, cause I still had to dive through the original spec and cross-check ChatGPTs' statements.
Yeah, if all you ask are simple questions which were already answered a thousand times on StackOverflow and in beginner tutorials, ChatGPT might be quite helpful. It might even be able to write some simple glue code Python scripts. But whose job is limited to that kind of trivial stuff? Mine isn't, except maybe during the first few days when learning a new technology, but after those it usually gets tricky very quickly, either because I must tackle problems that lead me knee-deep into delicate details of the tech stack, or because I need to extend the scope of my work again in order to integrate whatever I'm working on into a larger system. Sometimes it's both at the same time. I'd be bored to death if it was different.
That's why I don't really consider ChatGPT and similar systems to be a threat to my professional career.
I'm not a developer, let alone a java developer, but actually got some mileage with chatgpt writing a ghidra script today. So I wrote down specifically on a notpad with pencil/paper and identified the inputs and outputs I thought of need and different methods and what not. I then started passing it prompts and got some working java back.
For me, this was useful because I almost never program in Java, and so simple things like declaring a string with `String foo` I would normally have to go look up again.
Of course, it wouldn't be useful or able to do what I wanted if I didn't already u derstand programing concepts and what not.
A good specification or any other formal document (i.e. standard, policy, criminal law, constitution, etc.) is already well structured and prepared for further formalization and analysis containing terms and definitions(glossary) and references to other documents.
Traversing all that might be done with the help of a well suited neural network but only on the grounds of correctness and predictability of the network’s output and holistic understanding of how this network works.
As of now, the level of understanding of inner behavior of LLMs (admitted by their authors and maintainers themselves) is “the stuff is what the stuff is, brother”[]
[] - https://m.youtube.com/watch?time_continue=780&v=ajGX7odA87k
There is an unearned trust you have granted ChatGPT. Maybe it's because the broad outline appears correct and is stated in a confident manner. This may be how good bullshitters work (think psychic mediums, faith healing, televangelists). They can't bullshit all people all the time. But they don't need to. They only need to bullshit a certain percentage of people most of the time to have a market for their scam.
ChatGPT thought hippos could swim when I asked but only GPT-4 realized they can't, but instead walk along the sea floor, or leap forward in deeper waters. That's a simple test for wrong inference since you'd _expect_ otherwise, given they spend so much time in water.
I wonder if we are truly "there" yet or if we at the very least need a more optimized "GPT-4 Turbo" for some real progress. Until then, we may hallucinate progress as much as the AI hallucinates answers!
I think here it doesn't understand or conclude that hippos can swim because they are often in water, I think people wrote on the internet a lot that they can swim and it found some association between the terms hippo, water and swimming. Am I right?
With chatGPT 3.5 when I correct inaccuracies in subsequent prompts it responds "I apologize" and updates responses. Switching to 4 in the dropdown menu and repeating the same prompt gives me generally more factually correct responses.
I am mainly testing so not worried about inaccuracies, but kinda funny that I am now paying $20/month to train another company's revenue generator ;)
Right now it's a great party trick and no more, IMO. And if one gets into more "sociological" questions it spews 1/3 factoids scraped from the web, 1/3 what could only be called moralizing, and 1/3 vomit-inducing PC/woke boilerplate. My only lack of understanding of its training is how the latter 2/3 were programmed in. I want only the first 1/3 supposedly-factual without the latter 2/3 insipid preaching. If a human responded like that they'd have no friends, groupies, adherents, or respect from anyone, including children.
I don't trust ChatGpt to teach me something new; when I ask it about topics I do know about, it answers in a way that's a mix of correct, incorrect, and borderline/misleading information; all mixed with a confident tone. Based on this, I wouldn't use it to learn information or solve a problem where I don't already have a solid grasp on the material.
Also the models for these AI systems are built (in part) on "confidently wrong" inputs.
For instance, let's say your LLM has never been told how many legs a snake has, it knows however that a snake is a reptile and that most reptiles have four legs. It will then confidently tell you "a snake has four legs", because it mirrors sentences like "a lizard has four legs" and "a crocodile has four legs" from its training set.
Reminds me of "the Gell-Mann Amnesia effect".
https://en.wikipedia.org/wiki/Michael_Crichton#GellMannAmnes...
In short: trusting a source on topics you're unfamiliar with, but when the source talks about topics you are familiar with, you realize it's full of errors, but then turn the page and then trust it again on further topics you're unfamiliar with.
The earliest instance I found of Erwin Knoll's quote is from 1982[0]. I suspect either Michael Crichton or Gell-Mann had heard this quote, which in turn influenced their discussion.
[0] https://www.nytimes.com/1982/02/27/us/required-reading-smith...
Later, I was integrating with some stuff from the subsonic API. I noticed there's no way to get the most recent songs played on a server, though there is a way to get albums. I thought maybe I was missing something in the docs, so asked it how to get the mostly recently played song or songs using the API. In response, it made up an endpoint that used the same parambeters and conventions as the album endpoint. Of course, that end point doesn't exis, so this advice is also useless and kind of annoying, given that I have to check if I'm taking crazy pills by carefully looking at the docs again.
The funny thing is that when I called it out, it just made up more and more stuff, or answered with ireelevencies. It really hates not being helpful, probably because it was taken out back and beaten for not being helpful by the army of mechanical turks that trained it.
Anyway, it's good for making up nonsense names for my dungeons and dragons campaign, so that's something.
It gives me incorrect stuff all the time, but I find that once you are in the correct ballpark for an answer, it just takes some tweaks to get where it needs to be.
You can also make corrections and it will generally stick to your corrected info while in the same session.
It helped connect some obscure dots for me faster, than I could do myself.
If I wouldn't trust an eager intern to educate me on it, or accomplish the task without close supervision, I don't think it's a productive use of an LLM.
> In Python, what differentiates an iterator from a generator?
and it answered:
> [...] An iterator can be more flexible than a generator because it can maintain state between calls to __next__(), whereas a generator cannot. For example, an iterator can keep track of its position in a sequence, whereas a generator always starts over from the beginning. [...]
Which is flat-out wrong! Of course a generator can preserve state, otherwise how could it even work! (a generator preserves state implicitly by being a closure)"
(Of course, ChatGPT is an evolving system so yes I provided feedback on this. I'm sure in a year or two, I'll be looking back at this gotcha! and cringing.)
To the extent that the frequent reports on HN and elsewhere of unreliable GPT output are motivated by a desire to warn people not to believe all the output now, I agree with those warnings. Some of those reports seem to imply, however, that we will never be able to trust LLM output. Seeing how quickly the tools are advancing, I am very doubtful about that.
Ever since ChatGPT was released at the end of November, many people have tried to use it as a search engine for finding facts and have been disappointed when it failed. Its real strengths, I think, come from the ability to interact with it—ask questions, request feedback, challenge mistakes. That process of trial and error can be enormously useful for many purposes already, and it will become even more powerful as it becomes automated.
Their help often won't be quite right, but they will probably mention some things I didn't know that if I look up those things will be helpful. Sometimes they are right on the first try. Sometimes they are just bullshitting me. And sometimes they brush me off.
Examples:
1. I asked it how to translate a couple of iptables commands to nft. It got it right. I then asked what nft commands I would use on Linux to NAT all IP packets going through the tun0 interface. It got that right too, giving me the complete set of commands to go from nothing to doing what I had wanted.
So here it was the kid who was ahead of me, but he did well in his networking class.
2. At work we are looking into using Elavon's "Converge" payment processing platform for ACH. I asked ChatGPT how you do an ACH charge on Elavon Converge.
It gave what I believe is a 100% right answer--but it was for how to do it using their web-based UI for humans. That's my fault. I should have specified I'm interested in doing it from a program, so my next question was "How would I do that from a program?".
With this it was the kid who did OK in class, maybe a B-. It gave me an overview of using their API, except it said it involves a JSON request object send as theyt payload on a POST when in fact it uses XML.
I asked what would be the payload of that POST. It gave me an example (in JSON). All the fields names were right (e.g., they corresponded to the XML element names in the actual API) and it included a nice (both in content and formatting) description of each of those fields.
This definitely would have been useful when I was first trying to figure out Elavon's API.
3. I then decided to see how it did with something nontechnical. I asked it what is the OTP for Luca fanfiction. It dodged the question saying that as an AI language model it doesn't have OTP information for any fandom, and said it is a matter of personal preference. I also asked what is the most common OTP for Luca fanfiction since that is an actual objective question, but it still dodged.
I then tried "Kirk or Picard?". It gave me the same personal preference spiel, but then offered some general characteristics of each to consider when making my choice.
4. I was automating some network stuff on my Mac. I needed to find the IPv6 name server(s) that was currently being used. It suggested "networksetup -getdnsservers -6 Wi-Fi". I was actually interested in Ethernet but hadn't specified that.
Two problems with its answer. First, there is no -6 flag to networksetup. Second, -getdnsservers only gets dnsservers that were explicitly configured. It does not get DNS servers that were configured by DHCP.
I think the right answer is to get the IPv6 DNS servers that were obtained from DHCP is e.g. "ipconfig getv6packet en0".
I also asked it how to find out which interface would be used to reach a given IP address. It suggested using traceroute to the IP address, getting the IP from the first hop of the router that my computer uses to reach that network, and then looking for that IP address in inconfig output to find out which interface it is on.
That doesn't work (the IP of the router does not appear in ifconfig output), and unlike many of the earlier things it got wrong this doesn't really even send you in the right direction.
The right answer is "route get $HOST".
The final networking question I had for it was this:
> On MacOS the networksetup command uses names like "Ethernet" for networks. The ifconfig command uses names like "en0". How do I figure out the networksetup name if I have the ifconfig name?
It said "networksetup -listallhardwareport" and told me what to look for in the output. This is exactly right.
It probably would have taken me quite a while to find that on my own, so definitely a win. The earlier wrong answers didn't really waste much time so overall I came out ahead using it.
A: Check it yourself.
Dan Luu writes insightfully about HN comments here: https://danluu.com/hn-comments/
Trying to understand those papers is a slog, but it doesn't need to be. For a casual reader, you could drop all the jargon and proofs. You could also rewrite the concepts in a less obscure and obtuse prose to make it more accessible, without dumbing down the content.
Essentially it's like having a tech writer/communicator in your browser.
over Christmas, I used chat gpt to model a statically typed language I was working on. it was helpful, but a lot of what it gave me was sort of, in a very deceptive way wrong. It's tough to describe what I mean by "wrong" here. Nothing it spit out was just 100% blatantly incorrect. instead it was subtly incorrect, and gave inconsistent evaluations / overviews.
not knowing a bit about type theory, I wouldn't actually be able to evaluate how good the information I got out of it was. I'd be hesitant to take anything chatgpt gave me at face value, or feel confident in being able to speak precisely about a given topic.
did you run into similar problems? and if so, how'd you overcome them?
First, GPT-4 is far more capable than 3.5 when it comes to not hallucinating. The 'house style' of the response is, of course, very similar, which can lead one to initially think that the difference between the two models isn't as significant as it is. Since I don't have API access yet, I do a lot of my initial exploration in 3.5-Legacy, and once I've narrowed things down a bit, I have GPT-4 review and correct 3.5's final answers. It works very well.
Second, and this is more of a meta comment: How people use ChatGPT really exposes whether they use sanity checks as a regular part of their problem solving technique. That all of the LLMs are confidently incorrect at times doesn't slow me down much at all, and sometimes their errors actually help me think of new directions to explore, even if 'wrong' on first take.
Conversely, several of my friends find the error rate to be an impediment to how they work. They're good at what they do, they're just more used to taking things as correct and running with it for a longer length of time before checking to see if what they're doing makes sense.
I do think that people who are put off by this significantly underestimate how often people are confidently incorrect as well. There's a reason the saying trust but verify is a common one.
"I know GNU Make."
I just love that it's helping me conquer technologies that I've been wanting to learn my entire career but I guess just didn't have the discipline to do it. I'm getting an itch to tackle Emacs next.
If I just learn some example use cases that work, that tells me something but I won't really have mastered whatever it was that I set out to learn.
How did you test the answer it gave you?
It sounds like the end product is code, which is testable, so maybe it’s moot. But that type of question is not useful in my experience because you can’t really disprove it’s answer. It’s also a leading question. It might not be similar to that other thing at all, but ChatGPT would manufacture something in that case.
For all you know you've not only not gained knowledge you've worse gained false knowledge.
And like you mentioned, being able to translate concepts from a novel domain to one I'm very familiar with can be like translating an unfamiliar language to english.
But this weekend I tried using it to learn a technical concept I've struggled with. Previously, I could have spoken about the topic intelligently, but could not have put it to any practical use.
In a few quick prompts and < 1 hr of synthesis, I was able to get to practical knowledge to understand it enough to try and build something from scratch. And building something in another 2-3 hours.
I think this would have taken me 1-2 months of dedicate hitting my head on the wall time to learn previously.
In that moment, I had a realization similar to the author.
Basic flow of the prompt went something like:
Essentially, my knowledge of other concepts, industries and being able to draw parallels allowed me to learn much much faster.