I would love to know this too. For me it’s involved too much manual copy-pasting of existing code for context, for it to feel like it’s doing much for me.

For cases like that, copilot (with chat for context) might be more of what you’re looking for. Chatgpt specifically, I’ve been using for very light context / general tasks that I modify. I always consider the trade off between how much time I’m saving by having to write the prompt full of context.

I'd love to understand this too - my experience has been that I can generally write what I want faster than figuring out what prompt will get something close to right, and then editing/revising it to make it right.

Add to this the limited usefulness for generating code that's contextual - making some method deep inside a component tree that needs to reference a service class, and pick some dom elements to mutate etc... it requires knowledge and reasoning about the project and overall code structure.

I don't understand how folks are using it as a productivity booster, unless maybe as something like a better StackOverflow?

I have a bunch of examples myself. Here's a good recent one (prompts are linked about half way down the post): https://simonwillison.net/2023/Aug/6/annotated-presentations...

A few more:

- "Write a Python script with no extra dependencies which can take a list of URLs and use a HEAD request to find the size of each one and then add those all up" https://simonwillison.net/2023/Aug/3/weird-world-of-llms/#us...

- "Show me code examples of different web frameworks in Python and JavaScript and Go illustrating how HTTP routing works - in particular the problem of mapping an incoming HTTP request to some code based on both the URL path and the HTTP verb" https://til.simonwillison.net/gpt3/gpt4-api-design

- "JavaScript to prepend a <input type="checkbox"> to the first table cell in each row of a table" https://til.simonwillison.net/datasette/row-selection-protot...

- "Write applescript to loop through all of my Apple Notes and output their contents" https://til.simonwillison.net/gpt3/chatgpt-applescript

I dunno what you do for a job but if that is your hard problems or even medium problems then I think you lucked out big time.

I felt the same, SWE with 20 years experience. Absolutely none of this is relevant to anything I’ve been paid to do for my whole career.

If someone on my team was doing things this unusually they’d probably be let go.

I use ChatGPT for similar. I can spend time looking into various languages/libraries I don't use often and figure it out. Or I can ask an LLM and have an answer I can validate in the same amount of time I start browsing search results.

You're missing the point. These LLM which are replacing "junior developers" aren't what's going to be used for "hard or even medium problems" because you don't just give junior developers "hard problems" to solve in isolation. Now try to give some thought as to the types of problems junior developers are suited to solve. Now apply ChatGPT to those scenarios.

I never said these problems were hard.

But... they're things that require research. Do you know how to loop through all of your Apple Notes using AppleScript off the top of your head?

That research isn't free: it takes time.

As someone who hasn't done much work with AppleScript before, I would guess it would take me about half an hour to figure out how to do this. But there's a risk that it might take longer.

So the sensible thing is probably not to take on that project at all! I don't care enough about solving it to invest the research time.

But the time taken to chuck a prompt through GPT-4 and then test the results to see if it works is less than a minute.

I wrote more about how this is encouraging me to be more ambitious with my projects here: https://simonwillison.net/2023/Mar/27/ai-enhanced-developmen...

100% agree with this. I used ChatGPT earlier today to give me networkx code for computing the connected components of a graph and visualizing the graph. This isn't hard to do, but I don't use networkx all that often and I forget the exact API. I could go to read documentation and piece together an example myself, or I could ask ChatGPT for an example, which tends to be much faster than doing the former.

I Googled "applescript loop through apple notes" and this was the first result: https://apple.stackexchange.com/questions/225938/notes-scrip.... Looks exactly like what ChatGPT came up with for you minus the dialog.

I then Googled "applescript print to console" and got this: https://stackoverflow.com/questions/13653358/how-to-log-obje... ; admittedly this answer took a few minutes to read through, but the second most upvoted answer recommended using log, which your ChatGPT-developed solution (eventually) used.

This all took me < 5 minutes.

Yeah, those StackOverflow posts are the perfect illustration of why I find ChatGPT so much useful for this kind of thing.

Compare them to my original transcript. They don't provide me with enough information: how do I actually run that code?

I got ChatGPT to split me out a zsh script, which caused it to show me how to use "osascript -e ".

I think Simon has plenty of experience solving hard problems: https://simonwillison.net/about/

That's irrelevant, we are taking about solving hard problems with ChatGPT none of the examples are even slightly complex.

Not sure where this idea that my post was about "hard problems" came from.

I'm excited about using ChatGPT/GPT-4 because it takes the time I need to figure out how to do something "easy" down from half an hour to sub-5-minutes.

Dude, simonw invented Django web framework

They’re probably not hard and medium problems but if you can take care of these easy and medium problems with chatgpt, that saves so much bandwidth to have time to tackle the fun and “hard” problems.

Also, take a look at the problems and projects simonw actually tackles.

You don't get the point.

I work in Deep Learning Research. I don't get any help at all from ChatGPT for my core job. Copilot spews gibberish, too.

I do get enough help about peripherals. Some weeks ago, I needed help with Flask and HTML deploying a model to show it to stakeholders. (I learned Flask some years ago, but not needing it regularly, I forgot enough.)

The data cleaning, preprocessing, model training, making it better than humans were the hard tasks.

Deploying a Flask app with a simple HTML frontend was the easy task. But easy != free. It would have required 2-3x more time researching how to do exactly what I needed, which I did with Copilot and ChatGPT in ~1 hr.

sorry but these examples are not impressive at all and by no means a representative of any serious programmer's workload.

Programmers are paid not to bang out code, but rather to figure out the mess and crap of the existing codebase and how to selectively add one-two lines to change system's behavior and keep stability of the system.

Part of the trick of making the most of LLMs is figuring out what kind of things to apply them to.

I wouldn't expect much from them for figuring out gnarly changes in huge existing codebases (at least not yet).

But they've been actively encouraging me to write smaller, tighter tools - the good old Unix philosophy - for which they are extremely well suited.

This comment makes me feel as if I've been taking a wrong approach to using LLMs in my day to day work. I haven't been able to get much value out of them as a large majority of the work I do requires a good deal of context with some repo specific problems.

Our existing tooling and helpers lack modularity to begin with. I'm now thinking it would be a better approach to start with having it build smaller tools and patch them together in more useful and interesting ways myself as opposed to being upset that it can't deliver complex, context aware solutions.

Still not buying into the hype that LLMs will replace all software engineers by 2030 due to thinking the nature of our work is not to write code but to solve problems, regardless of the tools we are using, but I definitely see potential productivity gains from using the tool with a different approach to what I have previously attempted.

So basically llm is the new short shell script.

Fair enough, but i also don't really feel this is threatening anybody's job.

I think it's pretty useful to share concrete examples like this. I basically always have a ChatGPT4 CI tab open these days when I'm developing these days - it's a usually much faster/better go-to than Google or SO for looking stuff up. ChatGPT is great for all the random stuff I don't care to remember the syntax for.

* I'm always using it to munge/generated tables/csv/markdown/json - you can basically throw any copy and paste from a random PDF that's some weird gobbedlygook of tabs, spaces, newlines and get something cleanly formatted. On the one hand, it seems like a waste of computation, but on the other hand, it's way cheaper than my time and there are so many tasks that require using poorly formatted output. Even better, CI will of course write awk/sed for you if you need to do any automation.

* I'm always forgetting the syntax for named byobu sessions (it happily wrote a script to help with that) but I've also been staging some dev servers and it was able to generate the scripts to create new named session and windows, attaching/creating when necessary, handling if the processes were running, and creating the systemd units for spinning these up.

* On this same project it wrote some python scripts for managing SSH tunnels and reverse tunnels, including filtering/logging of error messages, handling jump servers, etc. This is all stuff I've done years ago (and even written lots of docs for), but it was actually way faster for ChatGPT to generate these than digging those out.

* I've been running into issues w/ some HTML5 audio output and needed to swap to websocket streaming w/ webmedia output (which I wasn't familiar with at all). ChatGPT gave me the code to swap into my FastAPI server and the frontend code I had w/o having to do any further research, great.

* I hate Docker setups, and I had issues w/ Nvidia containers and GPUs not showing up w/ my docker config. I was able pass the various error messages and get my problems fixed without spelunking/hair-pulling. Same with figuring out some cross-container network hijinx.

* There's a bunch of one-offs that I might just not have bothered doing, that I can just ask it to do as well - eg, I've previously written code for poisson distributions and the like, so I knew what to ask for, but would have been a huge PITA to dig out exactly how to do it, but took like no effort to just ask GPT4 CI to figure out a one off I just wouldn't have done otherwise: https://chat.openai.com/share/80fa7bc0-e099-4577-bad9-d026e7...

Those are good examples. I'm constantly doing weird one-off things in Ruby or Bash, and maybe I'll ask ChatGPT about more of them.

I really wanted to know what vague things OP's humans were doing, but they haven't responded to anything.

I've been reading your blog since the dark ages. Thanks for the great content over the years!

After putting some thought into this I think it has to do with the kind of developer you are. In my case I'm usually across 10-20 ecommerce websites doing various semi-unique jobs with relatively simple code.

Largely I use CGPT for work that's boilerplate/LOC heavy but architecture light, things like writing first drafts of React hooks and the like. It's quite good with constraints like use typescript or use X function to do Y.

I usually give it about two goes if it goes in the wrong direction on the first try. If it seems to not conceptually understand what I'm asking I generally just write it directly rather than tinkering with prompts for 20 minutes.

I also have a couple of longer system prompts saved for converting Vue components to React using the house style and things like that using the playground.

> but architecture light

It does fairly well for architecture, if you don't expect too many specifics. It, at least, works as a reasonable sanity check/brainstorm.

All of these LLM becomes less expert the finer resolution you take the context. Keep it high level, and you still have a relatively expert assistant.

My main use is TypeScript, which I am using for the first time and struggling with a bit. I'm fine with straightforward type definitions but I often hit complicated situationszi don't know how to solve. Googling doesn't really help because U don't know the abstract terms for what I want to do.

Instead I paste the JavaScript and tell ChatGPT to add type definitions. Mostly it gets it right. If it doesn't, it gets me closer.

I don't use it for JS in general because I'm particular about how I write stuff. Though occasionally I'll lean on Copilot to fill out a utility function.

yeah, I would like some examples that sent just trivial. I have failed to use it successfully for anything I cannot simply state in a condensed singular statement or paragraph. And almost none of my work is easily condensed into a single paragraph. Coupled with the complete misunderstanding it constantly seems to have and it's inability to understand nuance...I am struggling to make use of it and actually feel productive. Everything I use with it fails when I attempt to test it and it won't do anything complex because the tokens needed to explain the idea alone are quite numerous. I guess you could have it refactor code...?

The chat part lets you start with a base and you build on top of it, you don’t have to fit it all in one sentence.

For code I’ve found LLM mostly useless, since if I don’t understand something I need to read the docs anyway, and the generated code tends to be buggy even in react.

Where I have found LLM useful is in generating text. Where I used to use a thesaurus I now use LLM to find words to name things in themed UX. But it’s not great at function or variable names, it tends to pick names that look good but don’t precisely describe what something is. LLM is also great at generating text for role play.

I had a lot more success writing some code and have ChatGPT document it than doing the opposite. The documentation tends to be much better written than what I would have done by myself.

Indeed because ChatGPT is excellent at writing text. And because I know exactly what I want to see even if I have a hard time putting it into words myself, I can easily catch the mistakes and hallucinations.

I don't get why there is so much focus on code generating AIs and so little on code analysis. Have AIs do code reviews, write tests and analyze the results, etc... LLMs are awesome at reviewing code, they are able to tell you what's unexpected. And what is unexpected has a good chance of either being a bug or some key element of the code that needs attention. I think I have seen a single article about that, out of hundreds that are about code generation.

type a function signature and an opening squiggly brace, wait for "copilot" to autocomplete, press tab, ????, profit

if you use elasticsearch and not familiar with elastic search's syntax (I am not), you could use ChatGPT to write elastic queries for you.

same for SQL, if you are not familiar with SQL.

probably could be same with Splunk SPL, Kibana KQL, Prometheus PromQL, or any other DSL that you are not familiar with

the problem is that you dont get to know if the output is A-Okay. You just know "it works". This is the scariest part to me. Especially a DSL/programming language I'm not familiar with.

I want to contribute, while being fully aware what I'm contributing with. This doesn't lend itself to that.

I think that we are the silent majority. ChatGPT can be mildly useful for me as a dev, but in general it actually slows me down. There have been a few times when it has really shined, but it’s not the norm.

If it was actually a life altering tool (and it might be one day) there wouldn’t need to be an entire industry of people trying to convince everyone that with just one small trick Google doesn’t want you to know, you can quadruple your productivity.

It’s immensely helpful for learning. Orders of magnitude better than google which has ruined their search results with seo bait.

At the very least, its a much more powerful google (dont nitpick my comparison, i realize it hallucinates). Getting the EXACT context of your question is something generalized search/articles online will NEVER give you, and you can read hundreds of pages of docs all day. This is good for certain things, but not when you want to know just a single setting or atomic piece of information. I want to get the smallest amount of accurate information very specifically to my problem, as I'm programming upwards of 10+ hours per day on my own companies as a one man show.

My search history on chat gpt includes a few things as examples:

- specific ways SOLID applies or doesn’t apply to non-OOP languages

- helping me quickly learn nuances of Lua for configuring neovim, specifically for weird syntax or things annoying to google (ie what does # mean) or what does a specific error mean within the context of the configuration

- more efficient top k algorithms than what I was building for learning purposes

- asking to break down big o complexity of certain types of sort functions and whether they differ from n log n

- helping me learn enough rust to do a bug fix Pr that was annoying me

- x vs s in neovim config for keymap modes

- figuring out why Ruby doesn’t implement descending ranges

Etc etc etc

Google hasn't ruined search results instead the services like chatGPT are generating so much spam and crap that Google search can't really keep up. So the service you are using itself is making google and internet worse, and eventually that would turn chatgpt bad as well. Its a vicious cycle.

Google has MOST definitely destroyed their search results. They did this long before ChatGPT.

This is repeated on like a daily basis anywhere online, and lots of people agree. All you will see are basically fake amazon review websites, or affiliate links in everything. The top 100 pages are pure garbage that they choose to favor over actual websites with real reputation.

I've been coding 40+ years and I use it reasonably regularly, for things that are trivial time waster type stuff. Like I need some powershell command to automate something that I know will be automatable. Less so for coding, but I have a IDE AI code generator (Codeium) that is often good at predicting what you want to do next, especially for boilerplate type stuff. Then there's the times you are heading into unknown and just need a starting point, for instance I asked it to write a discord bot that did X Y and Z, and it pretty much gave me a good shell of a program. Didn't really have to refer to any other documentation. It's often good at finding ways to do obscure things. Quite often I find it most useful with TSQL stuff, not so much for basic queries, but there's lots of inbuilt toys I've just never come across nor care to spend the time researching. I can't see how it would replace a junior developer though. If anything, it makes it easier for junior developers to get up to speed.

Today I got chatgpt to generate a basic TCP server template in C for an app I'm working on. If I didn't have AI, I probably would have searched for a GitHub gist and there would have probably been a more accurate template.

I do Mac/iOS development and am constantly asking ChatGPT about various APIs and frameworks. Apple's documentation is not great for explaining how to actually use APIs, unless you can find the one WWDC video that explains it or a sample project that they released years ago. I would normally google for sample code, blog posts, tutorials, or Stack Overflow posts. Something that might take an hour of searching and reading now takes a few seconds of just asking ChatGPT.

Even for things that I've done before, it's often much easier to ask ChatGPT how to do something than to look through my projects to find how I did it previously. It might sound lazy, but if it takes me several minutes to search through various projects to find that one time I did something, why bother when I can just ask ChatGPT and know in seconds?

I will say that yes, ChatGPT can hallucinate APIs that don't exist, and that can be annoying, but even if it does it 20% of the time, it's still incredibly valuable in the time savings the other 80% of the time it does hit.

I wonder how this is affecting what you consider knowledge going forward. This strikes me as students using google to answer homework questions and forgoing the actual “learning” part.

I don't really think of it that way. I've been doing Mac and iOS development for over ten years now. A lot of the info I gleaning is not design techniques or stuff that I feel is worth memorizing. It's more what functions are available to do something and what types are needed to interact with an API.

A common thing I've searched for, for instance, is the various date formatting options and types I need for managing time zones. I suppose I could sit down and learn the plethora of options, but I don't see that is information that's worth memorizing internally. Similarly, I suppose I could really internalize the complete syntax of regular expressions, but is it worth it? I've used them so many times before ChatGPT, but I've not memorized absolutely all the options available to me.

The other side of the coin is this is allowing me to make so much progress that I'm needing to even use more APIs than I would've previously. If I had done it the old way, I might only have time and energy to devote to a small number of tasks, but with ChatGPT I can "explore" more territory than I wouldn't have previously.