I'd love to see the studies mentioning 10% efficiency gains if you have a chance. also, do these studies balance against cost of running the models? remember, that's why stability.ai has collapsed.

I am aware of this paper from National Bureau of Economic Research.

https://www.nber.org/papers/w31161

Would be curious how “software spend” is defined here. When people subscribe to Netflix does that go under “software spend” or “entertainment spend”?

Markets are just so tricky to define. The labor pool includes Netflix, even if they’re not competing for B2B SaaS spending.

I haven't seen LLMs do anything more useful than generating a wedding speech in the style of H.P. Lovecraft and there are people out there earnestly predicting this technology will increase efficiency 10% across entire economies. I genuinely feel like I'm living in a different world to you all.

I have yet to see an example of Copilot do something that doesn't seem like basic autocomplete/snippets that editors and IDEs have been doing for decades or code so basic that it betrays the lack of competence by the user.

I find the Copilot completions to be slightly better than IDE auto complete, but the latency is too high for it to be useful IMO. It's to big a break in concentration to wait for it to type almost what you were about to.

IMO chat-based question-asking when encountering unfamiliar language constructs or framework idioms is where LLMs shine in coding assistance.

> I find the Copilot completions to be slightly better than IDE auto complete, but the latency is too high for it to be useful IMO. It's to big a break in concentration to wait for it to type almost what you were about to.

This matches my experience perfectly. It gets decent/usable results like 40% of the time, but latency kills it.

> IMO chat-based question-asking when encountering unfamiliar language constructs or framework idioms is where LLMs shine in coding assistance.

I have never found this to be productive. It gets details wrong frequently enough that I can't trust the output which effectively doubles (or more) the cognitive effort because you have to validate that it's correct against a more reliable source, or find out the hard way that it's wrong when code/systems break.

Interesting, what systems are you asking questions to? I find GPT-4 is pretty much always correct about simple questions I ask. I just avoid using it for anything really deep and refer to documentation instead.

I've mostly used GPT-4 and Gemini Pro. For simple stuff, it can be correct, but it highly depends on the topic. I've found it very effective as a dictionary or thesaurus but for technical questions, even simple ones like small help with a Scheme macro, it's near useless. I imagine if I was asking it about, say, Python/Django APIs or basic HTML/webdev stuff, it would do better.

When you say technical questions do you mean writing code, explaining code, or something else? If it's writing code I agree, I don't often ask it to explain code unless I want to know about some obscure syntax, but it works pretty well for that. I would not trust it to explain complex logic in a function.

I've tried all of the above. I've asked it to summarize information about what languages in a class had strong symmetric multiprocessing characteristics and it made up over half of the details. I've asked it to generate code for a simple linked list and it created garbage. I've asked it to summarize what code does and it gets it blatantly wrong.

These are just a few examples. I've found it to be nearly useless for anything where a semblance of correctness matters. Every single thing it says must be validated against external sources and it can easily lead you down rabbit holes.

GPT-4 was a little better than 3(.5), but I've yet to see it be good at anything enough to warrant the price tag or the time.

The auto complete is much more effective than what IDE has...

Even if you type at 140wpm it is effective.

Do you not find the latency disruptive? I find that the auto complete results taking even a second and a half breaks my flow, so I disabled Copilot and I don't miss it

There are some UX concerns I wish could be improved, but I think I have been able to fit decently with the rhythm of it and know when it is efficient to use for me. I can't write multiple lines in a second. In some cases it will give me the exact 10 lines of code I wanted to write myself with few seconds. That is significant.

It should be better at auto importing and status indication I think.

Also it never makes typoes or rarely index+1 errors, things like that.

It is much less mental fatigue compared to if I have to make sure I did those things correctly myself.

I think the delay I kind of in my head use like a guessing game of what it will exactly give me or I will plan other things ahead.

Because it is also always interesting to see how accurate my guess for what it will give me is. Guess it is kind of fun.

It can write code. I can assure you that. Still need a developer but it can make a developer incredibly more efficient much much more than 10%.

He was saying that he expects to see a 10% increase in efficiency globally, not amongst programmers.

Have you used GPT4 as a drop in replacement for Google, especially while coding?

I don't have to constantly Google things while coding.

I didn't imply constant Googling. IDK about you but I frequently work with frameworks and tools I have not encountered before and I find GPT-4 to do a far better job of giving me quick and straightforward answers about how these tools work than Google. Not just one-off "how do I do X", or "explain this syntax", but also things like "how does framework X handle Y", etc.

I even use it to figure out how to find settings that are buried in cluttered UIs.

Even nicer

Nice

I think people will lose their jobs with all types of cost-cutting but the people left will be left doing more of the work. I think like Web 2.0 a lot of AI software will be created to manage other AI software.

Buster - Software that links databases and large language models

Vectorview - Custom LLM evaluation

Nuanced - Helps detect deep fakes and misinformation

> There’s a ton of terrible startups right now, but some of them will become whales. You can’t predict which ones ahead of time, so you invest in everything.

For what value of “you”?

To be fair, a lot of software spending in Enterprise is what I would like to call Bullshit SaaS, a la David Graeber's analogy. Stuff that was subscribed to by some PM once upon a time, but then later sat unused, even as the company gets billed on it annually or monthly. Not to mention a lot of SaaS simply buying and using each other's products, fully funded by VC capital (which is the YC model).

Take for instance any of the Talk to your PDF products out there. They were popular when ChatGPT came out, but nothing close to the billions of dollars claimed to be in that market. Chatbase made a few millions in revenue, before selling, but now that OpenAI has native functionality, I'm certain that product has been zombified.

Of course, this is different from the large enterprise-level products being built by, say, companies like Harvey AI, but the utility of those products within an org over steady state (after all the hype has died down) remains to be seen.