Maybe one day, we’ll be able to run it over all the Git histories, Jira tickets, Confluence documentation, Slack conversations, emails, meeting transcripts, presentations, etc. Until then, the humans will need to stitch it all together as best they can.
If your only aim is to use it like Copilot, sure, it's useful.
Just right after we have invented AGI.
so it's somewhere between 1.2x and 10x for me, depending on what I'm doing that day. Maybe 3x on a good day?
For example, just because the manifest.json worked doesn't mean it is correct - is it free of issues (security or otherwise)?
I would argue that every system in production today seemed to "just work" when it was built and initially tested, and yet how many serious issues are in the wild today (security or otherwise)?
I prefer to take a little more time solving problems, gaining an understanding of WHY things are done certain ways, and some insight into potential problems that may arise in the future even if something seems to work at first glance.
Now I get that you are just talking about a small chrome extension that maybe you are only using for yourself... but scaling that up to anything beyond that seems like a ticking time bomb to me.
And to be honest, I don't care for managers like that, so the feeling is mutual.
what are some ways to handle this XYZ problem. I see you might have missed sql injection attacks. would that apply here?
Same goes for code you find on the internet.
I got this out put for this line of code what do you think the problem is.
It may be able to regurgitate code for simple tasks, but that's all I've seen it get right.
But to see that you actually need to know in detail the things you're asking about.
After using it for some time I'm by now quite surprised when this thingy gets something actually right. But that are very seldom cases.
Even OpenAI says clearly: You should not, by any means, ask the AI any questions you don't know the answer already!
> more benefit out of GPT. you could ask it if it finds any vulnerabilities, common mistakes, other inconstancies. please provide comments on what each line does. what are some other common ways to write this line of code, etc.
And than it spits out some completely made up bullshit…
How would you know if you don't understand what you're actually doing?
That is, SERPs providing relevant discussion and exacting or largely turnkey solutions.
On “easy” tasks in technical niches I’m not familiar with, I would take gpt over DDG + SO the majority of the time.
I’ve had situations where I’m wanting a tutorial or walk through on something and mixing various sub stack and independent blog posts.
There is no consistency in quality or even correctness from those sources, while you must also deal with format and styling variation.
I block adtech within reason, but the problem of greyhat or SEO-focused filler also isn’t really a thing in gpt. You just get the fat of the land.
The biggest problem w gpt 4 is the cutoff date. The LLM needs to be updated regularly, the way Google initially seemed to crawl all the things and make them available to queries as they appeared.
Data recency and the ability to process excess tokens is going to show OP’s test as but a toy example of what the systems can do.
In the absence of constant updates of a massive model and all that entails, I foresee Companies temporarily dominating attention by providing pretrained LORA-like add-ons to LLMs at key events.
For example, Apple could release a new model trained on all of the updated Swift libraries coming to market at WWDC shortly after the keynote.
It can contain all the docs, example code and allow devs to really, really experiment and have warm and fuzzies on the newest stuff.
It could even include the details on product announcements and largely handle questions of compatibility.
If the companies hosted the topic focused chatgpt-like bots, they could also own all the unexpected questions, and both clarify and retrain on what the most enthusiastic users want to know.
This is going a bit of another direction, but I think all of this is very exciting and will hasten the delivery of software for brand new SDKs.
Have you tried out Phind? It's essentially GPT-4 + internet access. It hasn't been perfect, but it's been a very useful tool for me.
Also chances are great that the AI just spit out some code from that extension… (Of course without attribution. Which would make it a copyright volition.)
Definitely this. I spend about 10-15% of my time writing code so a 20% increase really doesn't save me a lot of time. Also AI generated code requires more reading of code, which is harder and more cognitively expensive than writing code.
https://www.youtube.com/watch?v=EZ05e7EMOLM
But an AI can't write higher level tests as it would need to understand large chunks of code (sometimes whole systems), which it can't.
Testing implementation details is always contra productive. Have you watched the video? (I'm not recommending videos often, as I think writings have more value per time-unit, but this talk is a kind of classic on that topic.)
Implementation details are in the eye of the beholder IMO. I'm open to reasons why that's not the case here.
As others have observed, maybe most have stopped using it.
Maybe Ray Kurzweil is kind of on the money about computer brain interfaces, that’s when the fun really starts.
But I think code has such a semantically useful structure that we should probably try and exploit that as much as possible before falling back to "just search for stuff that seems similar".
Check out the "future work" section near the end of the writeup I linked above. I have a few possible improvements to this basic "repo map" concept that I'm experimenting with now.
>Ctags generates an index (or tag) file of language objects found in source files that allows these items to be quickly and easily located by a text editor or other utility. A tag signifies a language object for which an index entry is available (or, alternatively, the index entry created for that object).
I added a few sentences of explanation and background on ctags to my writeup.
The main reason I started with ctags is because the `universal ctags` tool supports a ton of languages right out of the box. Each LSP server implementation tends to only support a single language. So it would be more work for users to find, install and stand up the LSP server(s) they need for their particular projects.
I plan to try some experiments with LSP in the future.
aider is a command-line chat tool that allows you to write and edit code with GPT-4. As you are chatting, aider automatically manages all of this context and provides it to GPT as part of the chat. It all happens transparently behind the scenes.
One interesting thing, it seems to work better to have the model figure this out in a way it understands, then have it use that. Still trying related experiments.
I’m eager to read your work in more detail. I haven’t documented much lately—everything is moving so quickly, mostly just tryin to keep tryin, before my last idea is someone else’s deployed product.
I still highly value our hacker collaborations, I think they’re essential.
I've been kind of tricking GPT, admittedly I'm currently limited to gpt-3.5-turbo, because with the API you can submit whole conversations and ask for the next reply from the assistant. I don't know how much of that whole conversation it re-reads, but I'll do things like:
- Give it a system prompt with guidelines.
- Give it my user prompt, which contains a description of what I want it to do.
- Give it an "assistant response" which I've faked which is like: "Can you tell me more about the database schema?"
- Give it my answer, which is the database schema.
- Give it another faked assistant response where it is asking me about the API endpoints I need.
- Give it my endpoints...
I've found that if I try to combine all the above information into a single prompt, I often run into token limits.
LLMs don’t understand. They don’t understand because they have no model of reality.
One may not simply claim that a machine ‘understands’ because it appears to. Now we’re actually in the much-misunderstood Occam’s razor: that the theory which introduces the fewest new assumptions is the most likely correct.
The assumption that a machine ‘understands’ — is capable of such a thing — is new, and requires extreme justification.
Yes. One can easily construct a conversation where it is obvious that ChatGPT doesn't understand.
I haven't seen any turnkey solutions yet good enough to use for development, but it's coming with months, not years.
> To address these issues, I send GPT a concise map of the whole codebase. The map includes all declared variables and functions with call signatures.
Even if you could encode a function signature in only one token (which isn't be possible, of course) this would only work for small single developer programs, and not large projects.
Mid-sized frameworks / libs have often teens of millions of lines of code, and hundred thousands of function signatures. With 8k tokens to spend you go nowhere even close to a large code base…
https://aider.chat/docs/ctags.html
The problem I am trying to solve is that it’s difficult to use GPT-4 to modify or extend a large, complex pre-existing codebase. To modify such code, GPT needs to understand the dependencies and APIs which interconnect its subsystems. Somehow we need to provide this “code context” to GPT when we ask it to accomplish a coding task. Specifically, we need to:
1. Help GPT understand the overall codebase, so that it can decifer the meaning of code with complex dependencies and generate new code that respects and utilizes existing abstractions.
2. Convey all of this “code context” to GPT in an efficient manner that fits within the 8k-token context window.
To address these issues, I am sending GPT a concise map of the whole codebase. The map includes all declared variables and functions with call signatures. This "repo map" is built using ctags and enables GPT to better comprehend, navigate and edit code in larger repos.
The writeup linked above goes into more detail, and provides some examples of the actual map that I send to GPT as well as examples of how well it can work.