Its a bit of a game changer if you already understand things so can point it to the right tasks and also clock that it's screwed up just from glancing at the code.

I haven't tried Cursor yet, I just get it to write or refactor functions bit by bit maybe that's better for monolithic tasks?

I should probably write a post about this, but comments are easier.

0) LLMs introduce the challenge of automated semantic verification.

1) LLM work should be broken up by usecase. These use cases lie on a semantic complexity continuum.

2) Tasks like classification are simple to verify (precision recall etc.). Tasks that require semantic complexity like summarizaiton are on the other end.

3) Anything on the high semantic complexity end of the scale needs expert human review.

4) Chained or complex calls greatly complicate verification

Which is why you are getting use out of it. There is a human (you) in the loop, and the calls are not complex.

As long as a human is reviewing the output of the LLM every time, its great. This is the vast majority of "generative" output.

Its when you have things like agents, or chained calls that things go awry. ITs why proof of concepts are easy, but production is hard.

For the record, many people have called this out, including people at Open AI, and AI ops was the largest sub group in YC's fall batch.

> I had hoped to be able to get complex tasks done entirely with assistance from the LLM. In the end it helped maybe 20-30%.

Ok but if you would do the same task again, how much would it help you?

> When I went back to the videos of people using chatgpt to build a website in under 30 minutes - its always someone who knows the domain extensively.

You are one step closer to become this person.

>Ok but if you would do the same task again, how much would it help you?

100%, since I would never do it again.

Was GPT4 out then? I self taught to build a Web app before LLMs, for the second one, I had LLMs and it helped me so much, in the area you mentioned: clarifying concepts and spotting obvious bugs due to inexeprience

Yup. I had Gpt 4, copilot, and midjourney to assist.

There is a difference when an LLM is used in limited ways, by someone who knows enough to discard bad output.

I'd put it this way - ChatGPT will help you build a ship in the desert without hesitation.

What model did you use?

I've been in the same boat. Earlier this year I wanted to up my game as well with an improved website and blog to go along with it, not even sure which frameworks to use.

Using ChatGPT has been a hit or miss experience, helping me maybe 30% of the time as well. But when it did help, it helped me massively, especially for quickly setting up Wordpress PHP designs and accompanying CSS. I can honestly say I couldn't have done my web and blog redesign if it weren't for ChatGPT. Not because I wouldn't have been able to get the knowledge without it– more so because I wouldn't have had the patience to figure all of this out in my spare time.

Using ChatGPT has certainly been a more fun experience than browsing documentation, but I did have to do the latter about half the time anyway.

I'm mind blown because it's already so good and I do imagine that it will get as good as you need it to become.

And yeah as you said with domain knowledge it works very well

I have generated some toy experiments for a monthish - gpt to generate code to do something - and did generate some web frameworks and pipelines - I did not have it generate a database - although that may be something to try next. I wanted to it generate circuit diagrams and molecules

My experiments https://www.qualitycontainers.io/ https://github.com/msrobot0/llmlexperiments/