It's a bit scary because knowing this stuff is sort-of our secret sauce and GPT-4 was able to give an even better answer than I was able to give. It helped us out a lot. We are now taking the solution back to the customer and will be implementing it.
A few additional thoughts:
1. I knew exactly what type of question to ask it to get the right answer (i.e. if someone used a different prompt maybe they would get a different answer) 2. I knew immediately that the answer it gave was what we needed to implement. Some parts of the answer was not helpful or misleading and i was able to disregard those parts. Maybe someone else would have to take more time figuring it out.
I imagine future versions of GPT will be better at both points.
I'm also not so worried about 'oh but the machines will keep getting better'. I mean, they will, but the above will still remain true, at least until we get to the point where the machines start to make and drive decisions on their own. Which we'll get to, but by that point, I/we will have bigger problems than computers being better at programming than I am.
- I want to create a Google Docs addon using AppScript which has an interface sidebar. Please describe the necessary steps to create a sidebar interface. The sidebar interface should have a title with the text "Google Docs JSON Styles", it should have a large textbox which can contain multiple lines of text (fill it with lorem ipsum), and it should have 4 buttons placed horizontally underneath the text box, with the captions "Save", "Apply", "Copy", "Paste".
This produced the necessary Javascript + HTML + external steps to set up the project.
- I'm building a website using a React frontend hosted on example.com with a Django REST API backend hosted on api.example.com - Authentication is handled via Firebase Auth, currently using an Authorization header provided for API calls. Now I want to have authenticated links such that the user can click a link to access an authenticated download via the API. What mechanism can I use to create a link to an API endpoint which will require the user to be authenticated?
This gave me a the frontend and backend code necessary to set up authenticated links, with several alternate approaches following further prompting.
- Given an owner id, repository id, commit id, and private access token; give me a URL that lets me download a ZIP archive of a Github repository.
In all of these examples I received the desired result with helpful descriptions.
As a counter-example:
- Write a program in Brainfuck that outputs the string "CHATGPT".
In this case, the result is the Brainfuck program for "Hello World", with a step by step break-down of the program explaining confidently why it would output "CHATGPT", while being entirely wrong.
Also Github Co-pilot has been good for a long time.
My experience:
- Copilot did a decent job at suggesting functions when I typed out comments. It got progressively worse as the compression algorithms got less common (eg. Huffman vs. entropy coding) The smaller the functions, the more manageable it was. <- You need to write good tests, you want to step carefully through every line of code it writes.
- ChatGPT got most things right, and a few things really wrong. It was always super confident. <- Super dangerous if you don't double check, basically only usable as a refresher on a topic you are already good on.
- If given a piece of code and asked "How would you suggest to refactor this?", ChatGPT gave mostly very useful ideas. <- This is something that I will keep in my workflow.
- The bigger the project got, the worse the overall codebase looked. It became a mix of styles, pretty inconsistent, more subtle bugs were introduced that took me a while to figure out. (The nice thing about lossless compression is that you know if the code is right by using it)
- My "productivity", how many lines of code "I" wrote, was way beyond anything I could usually reach. I do also feel like I got a very quick start into Golang with this, much faster and broader than reading documentation or doing a getting started. That said, that knowledge now definitely needs to be refined in order to use the appropriate concept at the right time. Comparing my code with more professional Golang code bases I spot a lot of things that need to be improved.
I follow the same steps as always to design the code, but when it's time to implement something, I ask the bot to do it, then I review it and move on to the next function.
I find it much more useful for soft things like writing a complaint to a vendor for me, dealing with customer service, or cover letters from a job ad that I only have to slightly tweak. Takes me five minutes instead of 30+ and I got a few compliments about my “outstanding” cover letter.
Also gpt4 >> gpt3.5 for coding.
Leftwise bitshift would make gpt4 worse though :P
I tried to make GPT-4 create an algorithm for an enhanced tic-tac-toe game (think Wordle vs Quordle to somewhat visualize the difference) and it's failing miserably.
Also it’s not that it writes perfect code straight away and you never have to re-prompt or change anything manually. But it’s still an insane speed-up.
You'll likely refute that as your mind is already made up, but there you go, another conflicting and confusing data point.
I'm sure ChatGPT4 would likely agree :)
This is not my subjective experience. It's better, sometimes somewhat, but also much slower. I'm not sure which one is the better tradeoff for real work. I have it on 4 by default now, don't want to think about this for every question I ask, but for my work, 3.5 was fine.