I keep reading responses like yours, but I haven't seen any specific examples of problems being solved, so it all sounds very abstract. In my interactions with ChatGPT, it felt like just interacting with a search engine. There were zero continuations between questions and responses, and nearly 100% of the responses contained incorrect information.

Edit 1: As an example, I just asked ChatGPT to implement the Warren Abstract Machine for me. It gave me two different implementations, both with caveats that they are simple examples and not the whole thing, and neither implementation even type checked. It feels just like reading someone's homework where they copied off of someone else's work and had no idea what's going on. I don't see the point in this if it's just going to give me some high-level idea, which I already have, and an implementation that isn't remotely complete, much less one that doesn't even run. "Additionally, you'd need to ... handle backtracking". You don't say, ChatGPT?

Edit 2: I've kept on asking it to implement things I already know how to implement, even things that it probably has my code (being it was on GitHub) in its training data, and it keeps giving me code that doesn't even typecheck. ChatGPT is just doing what I've always imagined it doing: it's just doing a statistical merge. It has zero concept of anything I'm asking or it's saying.

Edit 3: I asked it something regarding WebGPU. It gave the typical, oh that's complex and "Here's a simplified example using GLFW and WebGPU through the F# WebGPU bindings provided by FableGL". But FableGL isn't a thing that exists, and even if it did, it wouldn't have anything to do with WebGPU which has nothing to do with OpenGL. And it imported Fable.Import.WebGPU and Fable.Import.GLFW, neither of which exist.

So - I just reached out to Chat, and we started going back and forth, starting with "In linux, how many open sockets can there be at a time?". What's nice, about that wide open question, is you don't get a single answer, instead you get a briefing on Linux Sockets. File Descriptors, sure, but also memory, port range, TCP/IP stack limits, etc... It starts to lay out a roadmap towards solving the issue, answering the question you were interested in, rather than the one you asked.

I do a bit of back and forth on some scenarios, asking about /proc, and ss, etc.. seeing if I can track anything else out. And then, after spending about 5 minutes, and building context, I ask it "Is there anything else that can cause an error regarding too many sockets with low socket use" - at which point it lays out a number of scenarios, one of which is FD_SETSIZE.

So - we dig into FD_SETSIZE, and immediately that looks interesting - it's the limit that you can use with a select() call, and, even better, I get recommendations to use POLL or EPOLL (which anyone who has ever straced has seen a ton of).

I ask it how to determine FD_SETSIZE, discover it's 1024 on the client, which matches our low socket count, confirm that we should never increase the size of FD_SETSIZE, check the vendor code, see they've got it hard configured to talk with select() instead of poll - we recommend they give us a new build with poll() defined - and voila - problem goes away.

On to the next issue.

Where Chat excels is not in solving or completing anything, or, in fact, even being correct 100% of the time (it clearly isn't) - but it's an endlessly enthusiastic savant intern - frequently wrong in hilarious ways, but always willing to dig in and pull on technical threads and come up with things that I can try with a bunch of rapid iterations, and close off an issue. It's willingness to write code that is 90% correct just reduces the time, and cognitive load of constantly having to do it all yourself.

I’ve had to sharpen my skills at verification and cross-checking, but the volume of pretty good insight from a much wider domain than I myself could consider has been so worth it.

It’s also worth noting to that I feel like I used to get this type of help, roughly, from good search engines. Maybe not quite as spoon fed but pretty good. But now… just blogspam and paid listings.

"I've essentially got 15-20 high-priced world-class consultants in every field that I chose to pull from"

And then here said:

"It's an endlessly enthusiastic savant intern - frequently wrong in hilarious ways"

And those feel very different to me.

The second one feels more similar to my experience This is what I spot with ChatGPT (and advocates of it): in the abstract, it is a genius, but whenever we look at a specific example, it's not like that at all.

I've been doing dev for ~20 years now, most of the work out there is essentially plumbing events/databases/REST APIs/regexes/JSON/configs/ web UIs in existing and well established frameworks/tools to solve business problems.

You likely work more R&D style environments, which again, probably represent a very low % of devs out there

Grandparent specifically said they find ChatGPT useful for things outside their wheelhouse:

>> [...]troubleshooting situations that flow into me - working in areas that I have the slimmest of backgrounds in

It sounds to me like it may not be safe to use ChatGPT if you can't check its work.

There are millions of people making good use of chatgpt but some think it is a conspiracy theory and it actually doesn't work. What can be done? Let them enjoy their ignorance.

- Paste a bunch of log code about which I have no idea. Ask to identify and explain the problem.

- Wireshark / dmesg / OpenWrt configuration pasting, ask to fix the problem. For instance, I fixed a Wi-Fi issue in a heterogeneous setup, which turned out to be caused by a stray DHCPv6 server.

- Paste C code, along with an error log, and ask to fix the problem.

- Paste my program and a sample. Ask to extend my program.

- Proofread and format Markdown nicely.

- Paste government letters, asking for a response that includes <what I want>.

- Paste a chat log and obtain documentation.

- Paste a tax declaration, and ask to check for consistency.

- Paste my code and ask for critique.

When discussing versions, people often confuse versions 3.5 and 4. I am always referring to version 4.

Keep in mind that ChatGPT has been seriously and intentionally downgraded since March.

This debate frequently leaves me wondering if I'm encountering a coordinated effort by bots. The examples I listed above come very naturally to me. I can't understand why people don't try to paste whatever they're working on and check the results. If it's too complex, asking it to critique instead of create, because that's easier. It feels as though there's an effort to shape public opinion into viewing these tools as "immature" and suitable only for edge cases.

Stay classy. The rest of your comment would have been much better without this ad hominem.

I once spent an hour researching the responses on reddit and hackernews. Turned out if these profiles are fake, then they are really good. F.e. age is >5 years, they argue about sport teams and seem really human.

Basically this - even in a highly specialised field you would still find value in it if you're providing enough context and describing the problem properly.

People don't want to hear that they're poor communicators - it's much easier to blame the tech.

It's just hard to even know what to say to this. So you are implying that the problem lies with the human, who holds years of experience in education, problem solving, communication, etc., and not the "AI", which has no experience in any of that and has no real thought processes or problem solving or reasoning abilities? And people say ChatGPT is over-hyped!

I've had some chatlogs where people are not able to articulate even one clear sentence. They were fine telling me what they want, but not able to type the same sentence

This is so weird and seems to be a common thread. You, and others, seem to be saying that ChatGPT is not useful if you're too dumb or too smart. It just seems both tautological and strange.

My communication skills are fine. ChatGPT is just not useful for me, period. I've tried taking what people have mentioned here, trying to have a so-called dialogue with it when things it suggests, for much more simple things, aren't quite correct. It keeps suggesting all these things, but it just feels like an interactive StackOverflow, except all the suggestions are wrong or don't work.

I am switching to teaching / consulting in AI. Chatbots are a part of that. I've reviewed some chatlogs, and some people get abyssimal results, which are entirely explainable and are 100% their fault.

I found it getting worse for a bit but lately it's been really good. I would say it's being constantly updated, not necessarily getting worse.

It's not able to output complete code.

It always adds comments, even when I ask it not to.

It doesn't follow instructions in general.

In complex issues it has less "memory", that is it's not able to observe multiple constraints at the same time. The limit was much higher before.

Sometimes you have to ask it in very specific ways to achieve what you're wanting - it is definitely limited in some areas though.

https://chat.openai.com/share/64dc6384-1fc4-4506-9280-98dc8a...

Just take any code which is < 3 pages long. Ask it to optimize a function. Or review the code. It should give good results

EDIT: I had some deprecation warnings in python recently, which it quickly fixed. Googled that on a different computer and found https://stackoverflow.com/questions/76689364/how-to-fix-depr...

This is the transcript. Forgive my dumb last question, I didn't understand the code, what it did worked well.

---

User: Please fix the deprecation warning on this code: for i in range(0,3): df.loc[df['date'] < "2023-06-01", 'date'] = df['date'] + pd.DateOffset(years=1)

ChatGPT: The code you've provided updates the 'date' column in a DataFrame df by adding one year to each date that is earlier than June 1, 2023. However, this operation is performed multiple times due to the loop, which is likely not intended and could be the source of inefficiency or unexpected results.

To fix this, you should update the dates in a vectorized way without using a loop. Here's the corrected code:

  import pandas as pd
  # Sample DataFrame creation for demonstration (you should use your own DataFrame 'df')
  # df = pd.DataFrame({'date': pd.to_datetime(['2022-05-01', '2023-07-01', '2024-08-01'])})

  # Correcting the date manipulation
  df.loc[df['date'] < "2023-06-01", 'date'] = df.loc[df['date'] < "2023-06-01", 'date'] + pd.DateOffset(years=1)
  *This code will effectively update all the dates that are earlier than June 1, 2023, by adding one year to them, and it does this in a single pass without a loop. This should also avoid any deprecation warnings and improve the efficiency of your code.*

ChatGPT: Certainly! Below is the code block with both the original version and the improved version. The improved version eliminates the unnecessary loop and applies the date modification in a more efficient manner.

  import pandas as pd

  # Original code
  df = pd.DataFrame({'date': pd.to_datetime(['2022-05-01', '2023-07-01', '2024-08-01'])})
  for i in range(0,3):
      df.loc[df['date'] < "2023-06-01", 'date'] = df['date'] + pd.DateOffset(years=1)

  # Print the result of the original code
  print("Original Code Result:")
  print(df)

  # Improved code
  df = pd.DataFrame({'date': pd.to_datetime(['2022-05-01', '2023-07-01', '2024-08-01'])})
  df.loc[df['date'] < "2023-06-01", 'date'] = df.loc[df['date'] < "2023-06-01", 'date'] + pd.DateOffset(years=1)

  # Print the result of the improved code
  print("\nImproved Code Result:")
  print(df)
  *This code first runs the original version, prints the result, then runs the improved version, and prints that result for comparison. Note that the DataFrame df is reinitialized before running the improved version to ensure a fair comparison.*

Code: >>> import pandas as pd >>> # Original code >>> df = pd.DataFrame({'date': pd.to_datetime(['2022-05-01', '2023-07-01', '2024-08-01'])}) >>> for i in range(0,3): ... df.loc[df['date'] < "2023-06-01", 'date'] = df['date'] + pd.DateOffset(years=1) ... >>> # Print the result of the original code >>> print("Original Code Result:") Original Code Result: >>> print(df) date 0 2024-05-01 1 2023-07-01 2 2024-08-01 >>> >>> # Improved code >>> df = pd.DataFrame({'date': pd.to_datetime(['2022-05-01', '2023-07-01', '2024-08-01'])}) >>> df.loc[df['date'] < "2023-06-01", 'date'] = df.loc[df['date'] < "2023-06-01", 'date'] + pd.DateOffset(years=1) >>> >>> # Print the result of the improved code >>> print("\nImproved Code Result:") Improved Code Result: >>> print(df) date 0 2023-05-01 1 2023-07-01 2 2024-08-01

ChatGPT: You are correct. The two code versions do not produce the same result. The original code with the loop increments the date by one year for each iteration of the loop, but only for dates before "2023-06-01". This results in multiple increments for dates that are initially before this threshold.

In contrast, the improved code without the loop increments the date by one year only once, making it more efficient and potentially more aligned with what one might expect logically. However, if the intent of the original code was indeed to increment certain dates multiple times, the "improved" version would not be equivalent.

If the goal is to replicate the logic of the original code but without the deprecation warning and with improved efficiency, we need to understand the exact behavior intended with the loop. If the intent was to increment dates before "2023-06-01" by multiple years based on some condition, the improved version should be adjusted accordingly. ---

---

Also, although I haven't seen it before, the Warren Abstract Machine seems like too big of a job for GPT-4. It excels at smaller tasks such as "Convert this C code to idiomatic Rust", or "Write a SQL query that grabs records from X, joins with Y, filters by Z". You might need to make small adjustments manually or by saying "Rewrite it with this in mind".

One really neat trick it did recently is that I uploaded a chart png I found on the internet, and asked it to write Seaborn code that matched the chart's style. It took ~3 follow-up tweak prompts, but then spat out code that worked. It also handles things like "How do I adjust this label that's positioned in chart coordinates over by 10 pixels?", which is an absolute pain to figure out from the docs.

Especially since you're getting answers noting the complexity, I think you're just asking too much of it for now. Try smaller tasks for now, and wait a while to try the big ones again.

So in order to properly evaluate a service, we need to already be subscribed to it in the first place?

A mundane one: I have a list of lat-longs mapped to region codes, and I needed write a query to find the area of the convex hull of each region code. I knew how to do it in code, but I wanted a SQL answer. ChatGPT gave me the correct Redshift SQL.

A more involved one: I needed to maximize match x people y groups, where people can rank their choices up to 3, but as soon as they accept the match, they drop out of the pool. Plus a bunch of different constraints. Sounds like a stable marriage problem. ChatGPT proposed the Gale-Shapley algorithm, which was the exact one I was looking for.

This is because the client in question is in a regulated industry.

This week, we used it a bunch of times to help rephrase objectives while working in brainstorm sessions, allowing it absorb ridiculously large regulatory PDFs to give us summaries and ideas of what our greenfield project should focus on.

The code snippets tend to be bloated, but no worse than an intern would produce. A quick refactor, adding edge case handling and the code is ready to test and commit.

TL;DR don't ask it anything to do with logic, but anything to do with documentation and stuff like that it's pretty good

Things I’ve gotten value out of in the past week or two:

• Writing a job description

• Making a python script to automate some stuff in Asana

• Simplifying some management concepts so I could slack them to a coworker

All of these are things I could easily do myself. But with ChatGPT, they’re done 75% as well in 10% of the time and I don’t have to think hardly at all.

- Helping to refactor SQL

- writing jq commands (I simply cannot)

- writing shell code (it happens just infrequently enough that I can't justify spending the time to get good)

- brainstorming names or puns. Word association is easier with a second person (or an LLM)

- figuring out why my AWS CLI commands aren't doing what I'm expecting them to be doing

- asking for reasons why a piece of code wouldn't work

I can competently do all of these things on my own if I havw to, but now I don't have to do them on my own so my life is easier.

I’ve also used it quite a lot for advice as I was learning project management and moving away from individual contributions.

So I think as a consultant it has value, though I think “world class” might be a bit hyperbolic, since I still need to validate its output.

Can you expand on this? This is what I'd like to use it for, but I don't ever have any experience remotely close to helpful. The typical response is just something like "that's complicated but here's some gibberish that tries to show a simple example but is still wrong in ways it's not useful at all".

Do you have a specific example?

I get a half decent expert to solve problems I don't have the patience or experience to solve myself. That's literally the definition of what a consultant does. What you call "trivial" is what lots of other people describe what consultants largely do.

I could have looked up the individual parts and found the problems myself, but then I would have spent at least 15 minutes on it instead of 30 seconds.

I suspect it is because I tend to have off piste questions which are not recipes or high-level ideas. It is where it'd be most useful to me, but it is at the same time where least training data is going to be.

It is also where actual legitimate experts are most useful: the special sauce, not the meat and potatoes.

The main argument is that you can give it a block of real-world data like an email or code and take advantage of collective knowledge to identify outliers like bugs, bad grammar or incoherent writing which translates exactly to code semantics too.

I had searched all over the web for an example addressing my specific use case on google and couldn’t find one. GPT4 produced a working example for me and got me past that road block. I also use it regularly to suggest better coding patterns and I find it does a really good job at doing code reviews for obvious mistakes / anti-patterns

But my mind kind of works like: once I'm stuck I endlessly do my best to understand it, which sometimes mean that I grind to a halt. ChatGPT stops that entirely.

The fact that it always produces 2 to 3 bugs is fine.

I use ChatGPT 4, GPT-4, and Copilot every day. It is an "average intern" at many many things. Here's how I feel it helps me:

* Its interactivity lets me learn a lot (superficially) about new topics. I can then expand that with research of my own

* It helps me think outside the box or from other perspectives when thinking about e-mails, proposals, real-world scenarios

* When exploring a new language, framework or technology, it points me in the right direction.

* For quick scripts, using the code generation/analysis feature, if I direct it right (i.e. layout "the plan" beforehand and ask it to work the rest on its own), it gets a lot of it right pretty fast, saving me some time writing the code, figuring out the right libraries and the nitty-gritty details.

* It is great at giving ideas for why something might not be working.

Real things I've done with it:

* Discuss ongoing negotiations with clients, trying to better my proposal and better understand the clients point of view.

* Learn more about managerial or "business-y" topics, by allowing me to discuss things with it and iterate on that with my own research. It is a valuable "white board" to discuss with.

* Adjust my e-mails so they are more appropriate to the situation. This can involve changing the tone, shortening them, adding more detail, etc.

* In general, i've used it to find flaws in reasoning when dealing with people. For example, it has helped me question my own client proposals or approaches, by specifying where I was lacking (e.g. because I was vague, or pessimistic, didn't give a mensurable objective, seemed to ignore a client's request, etc)

* I use a command-line utility from the shell which lets me ask it to do something and then have it do it. I now use this with some frequency to just write the commands I would have to google because I haven't memorized. Things like ffmpeg or imagemagick arguments. Or combinations of grep, sed, ls, find, git, etc. Here are some examples:

i) "merge jpgs in this folder in order onto a pdf. each jpg should fill the page 100% without margin. Avoid complex calculations".

ii) "zip this folder with maximum compression and password 12345678'".

iii) "git remove branches matching pattern sprint-* remotely'"

iv) "use imagemagick to convert to jpg lossless and rotate 90 deg clockwise ~/home-en.png"

v) "Add _en before .jpg in all files in current directory. So they become _en.jpg"

vi) The list goes on and on...

* It has helped me cleanup nginx config files

* I have thrown code at it which I suspect has a bug. With a bit of context and some back and forth, it has helped me find it.

* In languages or frameworks I don't use often, it really shines. It has helped me write several applescript scripts which I have also cobbled together to create Alfred workflows. If I need to code something in a language I don't often use, what it produces is good enough for me to iterate on.

* It has helped people at our company improve their copywriting (when used with a lot of care)

* I have used it to help me critique my own poetry and improve how I approach poetry in general. Highly subjective, I know

* When trying to figure out how to use apps I don't often use, or dealing with unexpected behaviour in them, it often helps me find the issue. Notable apps in this category include Photoshop and Excel.

* I don't often do frontend so I'm particularly bad at styling and organizing things. When I ocasionally have to do frontend, it often gives me the initial "skeleton" very well.

I have seen many people try to use these tools, and here's where I think they SHOULD NOT use them:

* For facts, obviously -- which is unfortunately what many people actually try to use it for

* For writing (without check) most of your e-mails our posts, especially with very little context

* For ideas where you need "just enough creativity". It's a very fine line. Think brainstorming ideas for UI elements in a new website according to specific brand guidelines

* For incredibly specific details or overly complex tasks. You can't often have it write your 200 line function out of thin-air!

It is clear that GPT provides me with:

1. A better search engine, which skips past loads of outdated content or SEO-laden bullshit. It's just a shame that I can only use it this way for more specific "creative" or "problem-solving questions" questions. For important fact-based info, I always have to check what it says.

2. A partner with whom to discuss ideas and iterate over them, especially on topics I don't know. If anything, it's a great rubber-duck

3. A way to forget the underpinning of some of what I do and approach it with natural language. I find myself asking it to write my bash sequences instead of thinking them up myself. Of course I check them and 99.9% understand them, but it's just so much easier to have it do it for me.

4. (Copilot is an upgrade of my IDE. And a great one!)

If modern Google (and Bing and everyone else?) weren't so shit, I wouldn't need 1. To this day I still first go to google, eventually give up because it doesn't answer me properly and then go to GPT. It's ridiculous. I think 1. brings a lot of value to many many people. Google is so absolutely shit nowadays I can't believe it -- and the more I use GPT to get a comparison baseline, the more I am shocked. 2., 3. and 4. are incredible additions to my workflow.

I truly believe we are heading in the direction of interacting with computers via a new interface. Even writing code might eventually be affected. Perhaps, like we have built high-level languages on top lower-level languages and machine code, abstraction over abstraction, we might end up with some form of language writing which uses (very well controlled) LLM technology under the hood. Instead of directly writing the code, we tell it to write it. We still need to be smart to pick the right algorithms and structures, but no longer have to worry so much about writing the nitty-gritty syntax or details. Maybe?

Using GPT is its own art. Crafting the right prompts and getting a feel for how it works is very important. It is also essential to have critical-thinking and know when to question what it says and when not to.

A friend of mine told me this: "The companies who strive on hiring loads of shit people to do work are the ones who will suffer more. The others, which have hired smart people capable of critical-thinking, will benefit. GPT obliviates the work of many small shit people. Pair GPT with someone who is already smart and has very good problem-solving or critical-thinking skills, and now you've got a team that can obliterate the teams of many small resources.

I don't know if OpenAI will succeed or not, but I do know this technology is absolutely life-changing for many, and going back to a world without it being this acessible will likely be a net-negative.

But ChatBLT was ‘trained’ using SEO-maddened bullshit thus your getting the statistical mean of all that bullshit.

The first rule of the internet is don’t believe everything on the internet. Hitching your financial wagon to it seems crazy to me.

How? Maybe because it was trained on lots of data and therefore isn’t just “spitting the first thing”, or perhaps it’s because it’s actually just answering the question I asked, without muddling it in a page with 50 other similar questions designed to up the SEO ranking. Whatever the reason is, it clearly happens often.

I’m not advocating for not fact checking ChatGPT. Quite the opposite. But even with that caveat, it adds tremendous value, at least to me.

Would have taken me much more time to figure it out than what chatgpt did at first try.

I think it's worth stepping back here and re-examining the hurdles your setting for your own understanding.

The essential question here is: is ChatGPT useful to people. What you seem to be implying with your question is: I will not use ChatGPT unless it can solve problems with X level of difficulty for me. Why have you set that pre-requisite? Would it not still be useful to you if it simply increased the efficiency of solving simple day-to-day tasks that you're not blocked on?

Then I go on HN and read these paeans from other technologists who say ChatGPT has completely changed how they work and is 10x better than using Google. I'd like to have that too! I just don't get it, it doesn't match my experience at all. And yes I did try the paid ChatGPT model for a month.

This is exactly my experience and what I am trying to get at. I start feeling that maybe I'm missing out, so I go give it a try. My direct experience shows that it spits out gibberish, but then people say things like it's a godsend but don't give any examples besides maybe it generated some HTTP API requests for them.

GPT+ data analysis gives GPT4 access to a jupyter back-end.

I was dealing with a noisy sensor in a factory production line.

I asked GPT to remind me which averaging functions might be usable to smooth out the noise, within the constraint of a PLC with limited memory.

I got GPT to simulate the output from the noisy sensor, apply each averaging function, and supply graphs of inputs and outputs. It was then pretty easy to eyeball which function was most suitable for the task.

I then gave GPT a code-style example and asked it to provide an (IEC 61131-3) Structured Text implementation using that code style. Which it did. This turned out to be pretty close to the final implementation (after careful reading and testing.)

Because it's so cheap to do (time and money wise), I used GPT to generate quite a lot of throw-away code to get to the final result. I probably wouldn't have considered this particular approach if doing it by hand.

I'm still waiting for the concrete examples that back up the above commentary about how much of a gamechanger it is.

> I've essentially got 15-20 high-priced world-class consultants in every field that I chose to pull from, working at my beck and call

In other words, this comment from above needs backing up. I've seen such descriptions everywhere but no one has ever provided examples that corroborate this.

For me, personally, it has replaced Google Search for the type of query that looks like initial research on a specific subject. "What is X?" then ask a couple follow up questions and I can quickly get a rough idea of some subject that I happen to need at the moment.

I wanted to use my GPX files to generate my own private hiking/running log.

With basically zero knowledge specific of GPX files, D3.js, or mapbox before starting I was able to quickly write a plugin that displays a path on a mapbox map as well as generates an elevation profile.

It would have taken me so long to do this before by googling and reading documentation with trial and error I would have stopped the project.

Now I have a working system.

> > I've essentially got 15-20 high-priced world-class consultants in every field that I chose to pull from, working at my beck and call

I'll give you that: many of the claims are hyperbolic & ridiculous, including that particular one by the gp. The only way I can think of that statement being remotely true is if "high-priced" has no bearing on the quality of work of those consultants (though tbh that is often the case in reality).

Personally I have not found ChatGPT particularly good at anything I need to do. I have however found it very passable at many things I don't enjoy doing. It's reporting boilerplate is far more tailored & novel than anything I could ever put into a reusable reporting template. I dread writing reports: it's not a technical challenge for me but I gain no stimulus from it, so reducing the task to one where I just have to heavily edit something ChatGPT prepared for me is frankly incredible. The same goes for formal cold-call emails to people I don't know: another menial, unpleasant but necessary task.

It's also great at spitballing, and - as others have mentioned - quick shallow topic summaries of things you'd normally rely on a quick google & scan to get a quick topic summary (timely as Google gets worse and worse).

These are microgains throughout your day, nothing truly revolutionary, but that's the case for many successful tech we take for granted.

I was also able to use it to generate software to check the blockchain to gather data I needed, again with no experience using the blockchain client libraries

Can you give some example prompts that will reproduce such an experience?

This year I used GPT-4 to write a significant amount of Terraform that was necessary to migrate an application onto AWS.

Writing Terraform, in my opinion, is problem that's broad but shallow. GPT-4 needed to do little beyond summarize documentation, but it was able to do so competently, and that was hugely valuable to me.

Conversely: In my free time, I've attempted to use it for a game-development side-project, and very little of its output has been useful at all.

I can also feed it a list of web accessibility issues and have it sort the list into most/least critical, along with providing references to the specific WCAG criteria. It has occasionally stumbled on this task, but again massive timesavings.

I also use ChatGPT for filler text in designs. Yes I have to build on what it writes, but it's way better and quicker than what I'd do myself. I know our communications team is doing this x10 more than I am.

1) I needed to display a very high resolution image to a user. I have experience in GIS and imagery, so I knew I should use an image pyramid in some way, and from previous experience assumed I needed some server to cleverly serve the tiles. But I didn't want to implement it myself, and googling 'map server' led to rabbit holes. I consulted with chatGPT, and while it gave me several fake solutions, eventually it suggested using gdal2tiles.py to create the pyramid and then serve it directly with a CDN. This never occurred to me, and is a much better fit to this problem. This saved a LOT of time (on either building my own server or fudging with other solutions) 2) I have a streamlit service, and needed to use some of my infra inside, infra that was written using async-await. Unfortunately, streamlit and asyncio don't play nice together (boo streamlit). I went to chatGPT hoping it would find me a way to make it work anyway, and after trying everything it suggested (which failed), I tried googling myself, and spent several days without a solution. Eventually, I went back to chatGPT and it suggested building a small HTTP service that would serve the results, and access it in streamlit using requests, no async-await required. It's a hacky solution, but significantly faster than reimplementing my streamlit dashboard in another framework or rewriting my infra without async-await. It saved loads of time.

I think you'd agree these aren't junior level issues. ChatGPT definitely didn't solve every problem I came to it with, probably not even most, and even when it did I had to intervene signficantly. I feel the more experienced you are as a developer, the less valuable it is. But when you need to tinker in a field you're not proficient at, or if you need to brainstorm a solution to a tricky problem, it can be a great tool. I understand why many swear by it. It takes a while to learn what sort of issues are good to turn to chatGPT with and which aren't, and also how to phrase those issues

I mean it. I'd pay money to watch streams of someone using chatgpt to solve non-trivial problems.

For example, trying to get chat gpt to do something very simple for my work, like implementing a convolution it took me in circles and circles.

It gets the general idea right, sure. But it actually makes significant minor errors that ended up being more confusing than helpful.

I admittedly give it a vague question "where the signals overlap"

In context of the original thing I was trying to do, I was filtering a receive signal and trying to remove the delay from the convolution

I pasted it to GPT4, said "I think this code has a concurrency issue, can you find it for me?" and sure enough, it told me what line of code was causing the issue.

It is also great for doing things that I don't do often. For example, I almost never write bash scripts (like once or twice a year). Now days I just use gpt4 to write the script for me. For example a couple days ago I needed to walk through a folder full of files and pass them one by one into another program, and if that call was successful, append the processed file to a list of files that had been successfully processed. Trivial, but again, I never write bash scripts. GPT4 had it up and running in a few seconds.

I can throw a file at GPT4 and ask it to create mocks for my objects so I can go ahead and start writing unit tests.

Awhile back I had setup a new express server, but after I added my CORS config everything stopped working. I spent an hour debugging, no luck. I asked GPT4 what the issue was, and it noticed I had defined CORS on the wrong line (after I had started my server up). Oops.

GPT4 is also great at coming up with sample command line options for utilities. Sometimes good examples in man pages exist, sometimes they don't.

Honestly, I don't use GPT4 as much as I should!

In my example I said "I'm trying to translate my React Native app into multiple languages. I need an object that looks like this: <example JS code>. Can you generate this object for French, German, <insert more languages>, and for the following words: <words I needed>."

Luckily I know basics in a few languages and the words were simple enough that I could easily verify it wasn't a hallucination. I also know enough JavaScript to know that it was syntactically sound, but my IDE would've told me if it wasn't.

I remember the first time I watched a Netflix developer livestreaming their workday using an impressive neovim setup. It was eye opening.

I need the same experience for this ChatGPT fanaticism.

the goal here is to accelerate productivity, is it not?

It can be very helpful to guide you in a direction maybe you didn't consider looking into to begin with, but I don't fully trust it.

There was a fun example I had a few months ago where I wanted to see how well it would do with being asked to solve a problem iteratively instead of recursively, something like: "generate all valid, distinct permutations of a provided string given that you have a dictionary to check valid words against" and it got most of the problem correct, but when asked to fix anything it would go right back to a recursive solution with the same issue appearing, or in some cases a new issue.

It got me most of the way there with some edge cases I needed to handle myself, but it definitely seemed like that was as far as it was going to be able to go

Google Used to be able to handle a lot of the non-procedural questions - but somewhere in circa 2022/2023 something started happening to it's results and I started getting back mostly SEO churn, to the point I was going back to using manuals and having to dig in and learn the fundamentals on a lot of things - which is unsustainable if you are touching 30-50 different technology stacks.

Chat changed all that - I can now go 3-5 levels deep in some stack, ask some incredibly nuanced question, get a reasonable answer that points me in the right direction, close off the issue and then move onto the next one.

IMO, this is the crux of the problem, and it’s caused by allowing dev teams to use whatever stack they want, without also requiring them to support it. More cynically, it will always be the case if teams are allowed to use whatever they want, regardless of “service ownership” – because when there’s a SEV-1 and the owning teams have no idea what’s wrong, the SREs are going to be told to fix it.

Are you a manager?

My cynical take is that this happens when managers choose tech as they know nothing and push for LinkedIn driven development

Being given some code and reviewing it is a lot quicker than writing that code. Copilot is great. Half the time it spits out the wrong answer but you can see what it was “thinking”.

Phind giving you an answer and references lets you quickly double check. Sometimes it hallucinates but the answer and references combo is much better than a Google search which in turn is better than nothing.

Of course I recommend people not use AI for everything. I will go straight to MDN for any WebAPI question and use —help as my first port of call on the command line. This is like your L2 cache as a developer. Using AI for everything is like swapping to disk for everything.

Learn the limits of your tools.

Sometimes you need to go back and forward a bit, I tell ChatGPT it’s wrong and the error message and then it spits out the correct result, sometimes I need an algorithm tweaked because it has assumed a wrong constraint on the problem, again just explain it clearly and unambiguously and it will make corrections, there’s only been maybe 1/40 problems I couldn’t get a correct answer on after (sometimes a lot!) of back and forwards.

I am not looking for a perfect oracle, I am looking for something to write 80% of the code and then I’ll fix it up. It’s still way faster this way, especially in domains I don’t know, E.g. I just learned CUDA with ChatGPTs help.

It’s not perfect, and neither am I, but it doesn’t have to be perfect to be useful, you can get to millions in revenue through 80% solutions

Examples I can think of where it gave wrong info was:

- I asked to compare the size of the MacBook m1 and the 16" framework laptops

- I asked how to add the nixpgs-unstable channel in NixOS.

- I asked to help write a power shell script to create AD accounts.

The first one is open ended and I don't think M1 came before the "cut date of knowledge" in chatgtp. The 2nd one is too niche, likely not enough data to grab.

For the 3rd one, I don't know if the "date of knowledge cut" isn't recent enough - powershell became a thingy more recently and isn't wide adopted.. but I would expect to be a good fit

Claude Instant will not give you the highest quality answers.

It's like if an alien visited and the first place they went was to a convention for Special Needs people and then they decided that humans were too stupid to talk to. Or rather, there was a Mensa convention next door with a $20 entrance fee and they figured it wasn't worth it.

Or rather, for $20 everyone at the Mensa convention agreed to log in to a special app and wait 24/7 for any time you have a question. Then they would instantly select the most knowledgeable among them in that subject and immediately enter a reply incredibly quickly.

It’s written at least 20 python scripts with me and almost all of them have been close to perfect almost on first draft.

The real issue will be that majority of users will take the answer face value without even knowing if that answer is a good one or just nonsense.

"The AI said..." will create lots of issues. Time will tell.

Seconded.

Let me tell you what I used to do.

First, Imagine I have an error executing code or an error running some bit of 3rd party software. I go to Stack Overflow and search. I find posts related to my problem and I spend a great deal of time trying to shoehorn existing answers to my specific issue. Sometimes, my shoehorning works and I fix my problem. Other times, it doesn't work, and then I post on Stack Overflow myself. And I wait ... and wait ... for a response. Sometimes I get a response.

Now, when I have this type of problem, I tell ChatGPT, "Hey, I'm trying to do <xyz> and I'm getting this error <abc>. Help me troubleshoot this. And it almost always helps me fix my problem. And it's ~10× faster than Stack Overflow.

=============

Second, there are times where I have write code to do some relatively 'complex' data manipulation--nothing sophisticated, mind you, but stuff like, "I need these data columns rearranged based on complicated logic. And I need the text in columns A, X, AQ, and F merged, but only if <blah blah is true>. Otherwise, Just merge text in Columns A and AQ, except if the date in column ZZ is after January 1, 2019.". I can do this stuff on my own, but: a] it's cognitively draining, b] it takes time, c] I often make silly errors due to the complexity.

ChatGPT is, again, an order of magnitude faster than I am. And it makes _fewer_ errors.

It still makes errors. And I still have to know what to look for to catch those errors, but it decreases my cognitive load tremendously.

edit: I haven't used Stack Overflow in 6 months. And "Googling" is Plan B.

=============

Edit 2: I recently had to write a sympathy letter to someone whose husband died.

I knew the general ideas behind what I wanted to say, but I knew I wasn't going to write anything great.

I fired up ChatGPT and said,

"write a sympathy letter to <x>. Tell her that I didn't know her husband well, but the few times I met him, I could tell that he cared deeply about you<x> and his daughter. I know his daughter well and I think she gets a lot of her great qualities from him and you. Tell her I don't know what to say in times like this. Keep it short-ish. Avoid schmaltz and sentimentality because <x> isn't that kind of person."

It gave me about as perfect a letter as I could have asked for.

You just found your path to become a better developer than the majority then

I suspect this is one of the differences between people that find GPT helpful and people that don't--it seems much more potentially helpful when you are in a less familiar environment. In my work, when I run into unexpected behavior, if I knew what question to ask then I would probably already know the answer too. Until the day I can feed in an entire codebase and ask "What's wrong with this? vaguely gestures at everything", I don't expect that to change significantly.

So for an MBA major, 80% accuracy at 5% of the cost may be amazing but for me as an engineer and a person who cares, the inaccuracies are catastrophic.

I’m open to suggestions on how to work with this.

So, just like an expert consultant?

Two interesting things it did for me lately: insisted that 8 + 6 = 11 and also proved that P = NP. I don't know which solutions it may be providing you, but it can't be anything too complex, or at least not too abstract.

edit: typo

I think the "GPT is amazing" vs "GPT is useless" debate is just going to get more confusing as more versions are released.

LLMs are not good at search, math, encyclopedias, logic engines. Maybe some day they will be, but not yet.

For example, "I'm setting up a new TypeScript + Svelte app, I made some changes to the config that I thought were good, but when I try to run the dev server, I get this error." And then paste in your incomprehensible ten line error.

Any time you are using some software tool that you're not very familiar with, and you get an error message that you don't understand, try asking GPT4 to explain it to you.

It isn't so great at deep, theoretical, algorithmic questions. "NP reductions" are probably not a great fit.

Think of it as a research assistant that has a broad understanding of every technology in the world, but isn't as smart as you are about your specific area of expertise.

That would be amazing if Google, Microsoft, Amazon, or Meta sat on their hands while OpenAI got that big.

OTOH, I have found a use for stablediffusion that actually resulted to some income.