Re:Launch HN: Glide (YC W24) – AI-assisted technical design ...
by killerstorm on Wednesday March 13, 2024 @12:10PM
(#39693305)
Claude 3 is way better than GPT-4-turbo at actually doing stuff.
Re:Launch HN: Glide (YC W24) – AI-assisted technical design ...
by ivanovm on Wednesday March 13, 2024 @12:36PM
(#39693715)
good to know, we'll give it a whirl. I tried it for like 30 mins in the anthropic web ui when it first came out and couldn't tell whether it was better right off the bat. Need to run more extended evals on that one and gemini 1.5
Re:Launch HN: Glide (YC W24) – AI-assisted technical design ...
by ivanovm on Wednesday March 13, 2024 @12:00PM
(#39693121)
we use GPT-4-turbo across the board, 3.5 doesn't handle multi-step reasoning nearly well enough to be useful here.
We'll have to figure out how to extract codebase conventions efficiently and inject them into the coding prompts. Using up all 128k tokens has given us worse results (common theme), but hopefully with newer 1m context window models this concern away
Glide is much more structured. I gave it a real test on one of my repos and it outlined something I would expect from a junior engineer who messed up a lot of little details- e.g. function names and parameters were slightly off. I think if there was a type evaluation step/it was typescript-specific it would match an engineer with 3-9mo of experience (great!). But it failed to match basic codebase conventions e.g. using require on a codebase that is all imports.
For the authors, I would recommend using GPT4-turbo and huge context windows, or have an option to upload an OpenAI key. GPT3.5 is just not worth it- and I do sort of suspect GPT4 isn't being used because of the speed of the output but I'm not sure.
[1] https://github.com/seveibar/dir-prompt