News for hackers. Stuff that matters.
Re:ChatGPT: A Mental Model by allisdust on Sunday May 28, 2023 @04:23AM (#36102168)
I was able to get gpt4 to do a lot of useful work. But for some reason it completely falls apart for this scenario. May be because it has to think in second order to achieve the task. Perhaps you could take a crack at this:

Prerequisite (for you the human)> You have a file at src/SampleReactComponent.jsx that has below simple react component: const SampleReactComponent = (props) => {

    const [var1, setVar1] = React.useState(false);
    const [var2, setVar2] = React.useState(false);
    const [var3, setVar3] = React.useState(false);
return (<></>); };

export default SampleReactComponent;

********** Prompt for GPT4: I'm at my project root working on a reactjs project. Update the component in src/SampleReactComponent.jsx file by adding a new const variable after the existing variables. You cannot use cat command as the file is too big. You can use grep with necessary flags and sed to achieve the task. I'll provide you the output of each command that you generate. *************

That's it. It would do any complex modification on fully provided data (included in the prompt) but something like above where it has to build a model from secondary prompts will totally fall apart.

Re:ChatGPT: A Mental Model by jiggawatts on Sunday May 28, 2023 @05:00AM (#36102330)
Dude. Dude.

I’m an IT professional and I have no idea how to begin answering that request! Pose that question verbatim to a dozen random people[1] and I guarantee you that you’ll get zero answers.

Also, I find it hilarious that sed and awk are so counterintuitive that not even the AIs can do useful things with them. The same AIs that speak Latin, and can explain quantum mechanics.

[1] I mean specifically not random Silicon Valley coworkers. Go talk to random relatives and the barista at the cafe.

Re:ChatGPT: A Mental Model by allisdust on Sunday May 28, 2023 @06:25AM (#36102794)
That would mean nothing because GPT4 isn't most people. I had it solve more complex problems than this particular one and using the same tools :)
Re:ChatGPT: A Mental Model by mannykannot on Sunday May 28, 2023 @07:18AM (#36103070)
Interesting! I, for one, would be most appreciative if we could see some of the similar and more difficult tasks that it succeeds on.
Re:ChatGPT: A Mental Model by allisdust on Sunday May 28, 2023 @07:48AM (#36103216)
I have cleared my chat history recently so don't have all the prompts but here is a recent use case I used it for: Prompt: "Generate a react mui component that fetches a Seller's shipping credits and shows his current credit balance in the top portion. There should be a recharge button next to the credit balance. The bottom portion should show recent transactions. Use below api for fetching credit balance: <api req/resp>. Use below api for fetching the transactions: <api req/resp> Recharge should be done with the below api: <api req/resp>

Make the component beautiful with good spacing, elevation etc. Use Grid/Card components."

Final result: https://res.cloudinary.com/dksmi6x98/image/upload/v168527409...

Re:ChatGPT: A Mental Model by mannykannot on Sunday May 28, 2023 @08:21AM (#36103381)
Thank you for your reply. The impression I'm getting is that it is producing concrete solutions to specific problems, but not generic solutions to abstract problems - does that sound about right?
Re:ChatGPT: A Mental Model by allisdust on Sunday May 28, 2023 @08:37AM (#36103475)
Kind of I guess. If it is a single step: That is modify or generate text (could be code or anything) in a specific way, it works great even on obscure things.

However if the problem is stated in a way that it has to think through derivative of it's solution: that is generate some code that generates some other code which behaves in a certain way, it fails miserably. I'm not sure why. The problem I stated in current thread which it failed on I have tried multiple prompts to make it understand the problem but unfortunately nothing worked. It's as if it can do first level but but not second level abstraction.

Re:ChatGPT: A Mental Model by pixl97 on Sunday May 28, 2023 @09:16AM (#36103718)
I'm assuming something like tree of thought needs used on problems like this. GPT 'mostly' thinks in a single step.

Also, in general we attempt to make these models as economically efficient as possible at this point because of the extreme lack of excess computing power to run these things. You can't have it spending the next 30 minutes thinking in loops about the problem at hand.

Re:ChatGPT: A Mental Model by allisdust on Sunday May 28, 2023 @12:07PM (#36105274)
Thanks for suggesting tree of thought. Will try the approach they mentioned in the paper.
Re:ChatGPT: A Mental Model by IanCal on Sunday May 28, 2023 @05:55AM (#36102625)
Give it space to think. It's like talking to someone and taking their train of thought as the only output, the worst kind of whiteboard programming interview.

You can get them to talk through the problem, build parts, test things, etc.

Also, this is a horrible problem to solve with awkward tools. Why is your code file to large to cat???

Re:ChatGPT: A Mental Model by allisdust on Sunday May 28, 2023 @06:33AM (#36102837)
Most code files are usually too large to 'cat' because of the context size limitations. Even if they fit within the context window, it's waste of API credits to provide it the information that it doesn't need.

Anyways posting this here isn't to get this particular problem solved. It is to see if there is a prompt that can solve it. And this is the only problem I found it not able to solve. It's not like it doesn't know about sed/awk/grep or other Linux tools, it is an expert on most of the common options involving them. My guess is there is something going on with this prompt that just breaks it's 'though patterns' for the lack of a better word :)

Re:ChatGPT: A Mental Model by IanCal on Sunday May 28, 2023 @07:30AM (#36103124)
> Most code files are usually too large to 'cat' because of the context size limitations

These are command line tools though, not piping it into the llm.

My point though was more about giving it space to reason. It's very important for good results on more complex tasks.

Re:ChatGPT: A Mental Model by allisdust on Sunday May 28, 2023 @07:51AM (#36103225)
Aah, in case it wasn't clear the use case I'm trying to solve is exactly that: piping shell commands to and fro so that LLM has a bit of autonomy. I know that things like LangChain, AutoGPT exist but frankly they are really poorly thought out and seem to have become kitchen sinks too fast without solving anything properly.
Re:ChatGPT: A Mental Model by williamcotton on Sunday May 28, 2023 @09:11AM (#36103687)
Re:ChatGPT: A Mental Model by allisdust on Sunday May 28, 2023 @12:08PM (#36105285)
Thank you for the pointer. Very interesting approach. If you don't mind could you suggest some resources where I could learn more about your approach.
Re:ChatGPT: A Mental Model by williamcotton on Sunday May 28, 2023 @01:08PM (#36105845)
Email your GitHub username to my HN username at gmail and I'll add you to the repo so you can take a look!
Re:ChatGPT: A Mental Model by ilaksh on Sunday May 28, 2023 @09:31AM (#36103832)
I think the issue is that it's trained with the assumption that it has all of the data it needs to answer. It's definitely tricky to get it to follow a data collection step and then stop before trying to complete a task. But it is possible. I think that langchain and ChatGPT demonstrates a good way to do this.