News for hackers. Stuff that matters.
Re:GPT4 is up to 6 times more expensive than GPT3.5 by Maxion on Tuesday March 14, 2023 @06:02PM (#35159590)
GPT-4 is a scary improvement over 3.5, especially for handling code. It will be the literal definition of awesome when these models get a large enough context space to hold a small-medium sized codebase.

I've been playing around with it for an hour seeing what it can do to refactor some of the things we have with the most tech debt, and it is astounding how well it does with how little context I give it.

Re:GPT4 is up to 6 times more expensive than GPT3.5 by int_is_compress on Tuesday March 14, 2023 @06:17PM (#35159824)
There are already some cool projects that help LLM go beyond the context window limitation and work with even larger codebases like https://github.com/jerryjliu/llama_index and https://github.com/hwchase17/langchain.
Re:GPT4 is up to 6 times more expensive than GPT3.5 by Der_Einzige on Tuesday March 14, 2023 @06:22PM (#35159898)
The fundamental techniques that they use are highly lossey and are far inferior to ultra-long context length models where you can do it all in one prompt. Hate to break it to you and all the others.
Re:GPT4 is up to 6 times more expensive than GPT3.5 by hombre_fatal on Tuesday March 14, 2023 @06:39PM (#35160141)
> Hate to break it to you and all the others.

Jeez. Their comment is quite obviously a complementary one in response to the limitation rather than a corrective one about the limitation.

Re:GPT4 is up to 6 times more expensive than GPT3.5 by nico on Tuesday March 14, 2023 @06:35PM (#35160098)
Where can someone find and try ultra-long context length models?

Any links?

Re:GPT4 is up to 6 times more expensive than GPT3.5 by intelVISA on Tuesday March 14, 2023 @06:42PM (#35160197)
The longest one that is generally available is always going to be yourself :)
Re:GPT4 is up to 6 times more expensive than GPT3.5 by thelittleone on Tuesday March 14, 2023 @06:57PM (#35160440)
My context model is getting shorter and fuzzier.
Re:GPT4 is up to 6 times more expensive than GPT3.5 by EntrePrescott on Tuesday March 14, 2023 @08:10PM (#35161624)
… but still the weights are increasing ;)
Re:GPT4 is up to 6 times more expensive than GPT3.5 by faizshah on Tuesday March 14, 2023 @07:23PM (#35160922)
The methods they employ are to improve the context being given to the model irrespective of the context length. Even when the context length improves these methods will be used to decrease the search space and resources required for a single task (think about stream search vs indexed search).

I’m also curious what paper you are referencing that finds that more context vs more relevant context yields better results?

A good survey of the methods for “Augmented Language Models” (CoT, etc.) is here: https://arxiv.org/pdf/2302.07842.pdf

Re:GPT4 is up to 6 times more expensive than GPT3.5 by speedgoose on Tuesday March 14, 2023 @06:06PM (#35159658)
The 32k tokens version should do fairly well on such codebase. I don’t know if it’s the one used in ChatGPT.
Re:GPT4 is up to 6 times more expensive than GPT3.5 by Maxion on Tuesday March 14, 2023 @06:17PM (#35159829)
I’m not even kidding when I say I just did 6h or so of work in around 45 minutes
Re:GPT4 is up to 6 times more expensive than GPT3.5 by worldsayshi on Tuesday March 14, 2023 @06:42PM (#35160191)
Do you let it refactor your code base? How do you feed the code to it? Just copy paste?
Re:GPT4 is up to 6 times more expensive than GPT3.5 by Maxion on Tuesday March 14, 2023 @06:47PM (#35160272)
Yup, it’s a bit of a learning curve how to get it to output the most useful stuff with the least amount of prompts.

Figure out what your goal is, and then start by giving a wide context and at first it will give wrong answers due to lack of context. With each wrong answer give it some more context fro what you think the solution provide is missing the most. Eventually you get 100% working code, or something so close to it that you can easily and quickly finish it yourself.

This is one strategy, but there are many that you can use to get it to reduce the burden of refactoring.

Re:GPT4 is up to 6 times more expensive than GPT3.5 by intelVISA on Tuesday March 14, 2023 @06:48PM (#35160297)
I'm worried it's going to decimate the already wobbly jr market - who are the sr devs of tomorrow to wrangle the LLM output after it replaces the jr's job of coding all day?
Re:GPT4 is up to 6 times more expensive than GPT3.5 by visarga on Tuesday March 14, 2023 @07:21PM (#35160883)
All programmers that use AI just got a boost. I tried it and it is amazing, it writes complex code on demand like I never seen before. But competition also gets the boost. So now we have to compete with devs armed with GPT-4. Humans are still the differentiating factor, what company will use AI better?
Re:GPT4 is up to 6 times more expensive than GPT3.5 by theappsecguy on Tuesday March 14, 2023 @07:05PM (#35160580)
Certainly it will.
Re:GPT4 is up to 6 times more expensive than GPT3.5 by woeirua on Tuesday March 14, 2023 @06:56PM (#35160435)
The only thing holding this back now is lack of enough context. That’s the big nut to crack. How do you hold enough information in memory at once?
Re:GPT4 is up to 6 times more expensive than GPT3.5 by falcor84 on Tuesday March 14, 2023 @07:04PM (#35160577)
GPT4 supports 32k tokens, which I guesstimate would be ~25k code tokens and perhaps ~2.5k lines of code. So already enough to work on a whole module of code at the same time. If you're using small microservices, it might already be good enough.
Re:GPT4 is up to 6 times more expensive than GPT3.5 by IanCal on Tuesday March 14, 2023 @07:27PM (#35160987)
Also if you were doing it personally you'd probably not take the whole codebase, but look at function signatures/docs as you get further away from the code you care about. While there's probably benefits in clever systems for summarising and iteratively working on the code, you can get away with just cramming a load of context in.

I have been playing with GPT4 and it's good. It's easy to imagine a bunch of use cases where even the maximal cost of a few dollars ($1.80 ish for the context alone) for a single query is worth it. If it's good enough to save you time, it's very easy to cross the threshold where it's cheaper than a person.

Re:GPT4 is up to 6 times more expensive than GPT3.5 by tmaly on Tuesday March 14, 2023 @10:40PM (#35163066)
I can't get it to run a single query today. Are you on the paying plan?