> I'd say I'm getting 10x more non technical business advice in multiple areas than I'm getting with the technical stuff.

Probably in large part because you can't verify when its wrong since that isn't your area of expertise, so those advice looks better to you.

You should assume your org will be the same quality as a program written by a non-technical person using an LLM if you do it that way.

I was speaking to a fellow software engineer the other day. She has a tech lead on their team.

She reports that he is really nice, but some times he is insisting that he knows the working of some part of the system and is giving advice based on his understanding. Only thing is that he is wrong.

The software engineer needs to "de-hallucinate" him in order to get a proper answer.

I think we are at a time where we need not to categorically be critical about these things. Yes, preciseness is an issue. But there is a good chance that the LLM advice is on par with a random dude from a local accelerator.

The current generation of ChatGPT hallucinates far more systematically than that. The problem is that it can regurgitate what the random dudes from local accelerators have written in their blog posts. We all know blog posts are often a sales pitch and don't include useful advice, because you can charge for the useful advice you give.

> The problem is that it can regurgitate what the random dudes from local accelerators have written in their blog posts.

This is not how modern training pipelines work. The models are trained on proprietary alignments that ensure factuality.

Regardless, that random dude from a local accelerator probably regurgitates those blog articles himself.

"ensures", really?

Like with so many other thing, it is a process.

Just like reviews "ensure" correctness on Wikipedia - which is actually a good case. In my childhood teachers would make us avoid Wikipedia. Now they advice students to use it.

LLMs does compose answers from its corpus, it doesn't refer to it, so it doesn't ensure factuality at all since it can still compose questions in stupid ways just like it always did.

Most of the wrong thing it says is just invented by the model, not things it has seen, cleaning up the input doesn't add much there since it writes the same kind of bullshit as bullshitting humans do.

Yes, that is correct, and was also more than indicated in my initial comment.

> The software engineer needs to "de-hallucinate" him in order to get a proper answer.

The question is not whether a model is sound and complete. The question how LLMs perform in comparison to the existing solutions.

Don't let perfect get in the way of good.

With humans, it is often easy to establish who hallucinates and who doesn't. We can all identify a snake oil sales pitch. However, with LLM, it can often be a lot harder.

Let's take an example. I studied physics and can likely answer most questions on nuclear physics. However, as it is a while back, my answers will not sound as smooth, and I may have to correct myself a few times.

If you ask ChatGPT the same questions, it will come up with elegant, convincing answers. As a layman, you will likely prefer what ChatGPT comes up with, as it is answers are eloquent. Of course, until you build a nuclear reactor using ChatGPT knowledge. Once the fuel rods become critical and start making its way through the earth's crust, you may realise you should have relied on my stumbling answers instead.

That's a good point, but I think it's important to critically analyze any information you find online. Anything I read online might be bullshit, and yet I've learned a lot from internet. And many other people have as well. I personally see ChatGPT as a better and faster search engine. If I need something basic, it's often on point and almost instant. If I need intricate advanced details about a niche topic, yes, it can be wrong. So are other people. While I lack expertise outside of my main areas to verify it, I usually have enough to judge whether it's a basic knowledge for experts in this area.

At very least, ChatGPT doesn't intentionally try to mislead me to sell me something (yet).

I had the "pleasure" of working for someone who would take the AI's word over mine in my domain of expertise. It was a hair-ripping experience to say the least and I'm while I'm not there anymore, from what I can see they hadn't improved in that area by an inch...

Business ideas can be tested so this epistemic haze isn’t as much of an issue as you’re making it out to be especially when error rates are taken into consideration.

I have a friend who has business experience and has been getting verifiably helpful marketing advice from LLMs.

Sales are up. That’s all that matters for any set of instructions: does it lead to the expected end result.

These “do LLMs always hallucinate” questions are in the category of “do humans have a soul”. This is fine and dandy but the conversation doesn’t always need to go there as there is practical utility to discuss as well.

The same can be said for a human CEO.

> Probably in large part because you can't verify when its wrong since that isn't your area of expertise, so those advice looks better to you.

Which is a kind of Gell-Mann Amnesia. Which, I just noticed, has “LLM” in the name.

https://www.goodreads.com/quotes/65213-briefly-stated-the-ge...

>You should assume your org will be the same quality as a program written by a non-technical person using an LLM if you do it that way.

Only if OP is themself not somewhat business savvy

Way to ignore the first paragraph of the one you’re replying to where he says exactly this:

“ because you can't verify when its wrong since that isn't your area of expertise, so those advice looks better to you.”

I've seen a non programmer navigate bad llm advice on writing code by double checking the info, both with the same llm and with google searches. It isn't their area of expertise, but their light familiarity is enough to build something decent. I can imagine OP doing similar with business advice.

This is a critical insight. Whenever I ask an LLM about a topic I’m knowledgeable about, I can detect that it spews a load of nonsense.

Talking to people in other fields, e.g., history and literature, they say it’s the same for them.

LLMs are only useful if you know the topic you’re asking questions about. Otherwise, they generate so much nonsense that you cannot verify, that trusting it becomes a liability.

Depends what the topic is.

IME it hallucines rarely when asked practical programming questions but basically constantly when asked math questions beyond basic calculus.

The more abstract or nuanced something is, the less capable it seems to be unless you have a domain specific LLM trained on a corpus of quality data.

As a business owner with both applied experience and formal education on both the tech and the business sides, I wouldn’t trust random LLM-generated advice on business. There are nuances and a certain humanity in good business that an LLM is unlikely to pick up. And the Internet is filled with horrible advice and useless insight on running a business that it can’t distinguish from good advice and valuable insight.

My advice to anyone seriously considering a business is that a course or reading a good survey text on business will pay off in ways that ChatGPT can’t (yet?) give you.

It constantly invents functions that don't exist.

Deal with it like Wolfram, implement those functions! It's picking up on something. :D

It might be just a glitch but in the last few days it generates such blunders I suspect something must be going on. Like completely messing up random pieces of code on subsequent iteration. I don't remember anything like this before. Normally it would improve the code with each iteration (until going out of the context window and starting to "forget" things, but not putting random characters here and there on second reply).

Likely you are just moving slightly outside of its knowledge domain with your questions. Anything that is hard to Google will also typically generate bad responses, most coding problems are there but many programmers just work on the few common problems over and over.

> LLMs are only useful if you know the topic you’re asking questions about.

Not true, and you're missing a key point for the topic at hand. LLMs are useful if you know the topic OR if you have measurable outcomes. If you want to increase sales, and you ask an LLM how to increase sales, you will know if its ideas work if they result in increased sales.

Think of it like AlphaGo - it learned to play Go by playing with an AI despite having no knowledge of Go strategy, because it could determine what was effective based on outcomes.

But I would only know about the result for my sales in some weeks or months. I have to trust the LLM in the meantime - and your example implies that the LLM answer could be proven wrong. How is that different from asking a complete stranger about better sales strategies for my business? Or even throw a dice?

This seems like a good example of Gell-Mann amnesia as it applies to LLMs instead of journalism.

People are falling into the same trap of accepting authoritative sounding advice as true in an area they are not experts in.

The way I see the current crop of AI is that it has mastery of language and a very wide set of knowledge on many different areas but no deep understanding. Ask it about anything you are not very familiar with, and you will get something that looks like a really good answer. Yet, as soon as you try to poke it around subjects you do master, you quickly see its limits and shortcomings.

A few months ago, I tested its knowledge on precision machining, and not knowing much about the subject, I got what felt like really solid advice: I felt like it knew everything. Yet, when I reviewed what it told me with my dad, who used to work in the industry, he was not impressed: the answers were missing important bits of information or were sometimes plain wrong.

I do feel the current crop of AI is a game changer, but unless there is another breakthrough increasing its reasoning capabilities, I don't think human experts are at risk of losing their jobs anytime soon.

The current crop is disrupting occupations that deal purely with language: copywriters, translators...

Software engineers are probably very safe, but I believe that language specialists whose main skill is to deeply know a software stack will soon be hurting. I predict that software architecture and computer science skills mastery will become much more valuable than knowing a computer language syntax and associated API in depth. The AI will always be better than you at syntax now.

The place I see transformer-based models changing everything is human-computer interaction: voice and multimodality will soon become the main way humans interact with computers rather than mouse and keyboard, and this is huge.

However, I don't believe any GenAI model will be up to the task of running a company anytime soon, and I think that advice from a real CEO might prove immensely more useful than whatever GPT-4 is telling you.

GPT-4 is probably telling you the obvious—things you could have read on blog spam sites by googling your question. An experienced CEO would probably have unobvious and valuable insights that I don't believe you can get from ChatGPT at the moment.

Can you give a specific example?

I’m curious if it was useful because of the AI’s skills, or perhaps it was the conversation itself that helped, like in rubber duck debugging[1] or journaling.

In any case, interested to hear more details.

[1]: https://en.m.wikipedia.org/wiki/Rubber_duck_debugging

I find ChatGPT incredibly useful to learn the vocabulary of an unfamiliar area. Often basic information in the area I know very little about can be found easily once you know the right keywords. Even if ChatGPT gets something wrong, it does pattern matching well enough to tell me something like "yes, there is a well-known concept that matches your vague noob description, it is called X". It works well in CS where I have an intuition there must be a mathematical abstraction/algorithm that solves my problem, but I don't know the name. Once I learn the keywords, I sometimes find an open-source library for that or at least a high-level description of the algorithm.

It is also great at answering stupid questions. Sometimes I read a wikipedia article and don't understand enough to make sense of it. If I was in uni, I would have asked a prof: "Sorry, it is probably a very stupid question, but why X? and what is Y actually?" ChatGPT can give me ELI5 at pretty much anything and is faster than googling.

That is CS though; I’m not sure that would work with business advice, where problems can be less tangible or there can be no single or verifiable solution (nor library/algorithm). Unless this approach also helps in the context of business advice?

They're not necessarily good for advice, but you need a decent amount of business knowledge to run even a small business, and LLMs can help with all the noob questions you'll have if you're starting a company and never went to business school.

"What's the difference between amortization and depreciation?" types of things. You could Google them, but you have to waste time skimming through a couple blogspam articles, but LLMs can explain the concept in an easy to digest way much faster.

Are you using LLMs yourself like this in this context?

I've never used it for business advice. But it is still useful to learn expert jargon if you want to find relevant discussions. For example, I use it to learn about DJing recently and yes, often it answers "it depends on your personal taste and the party you are playing, you can choose to do X, Y, Z, etc". Then I know that there is no right answer. ChatGPT learned to handle such multimodality to an extent, often it doesn't want to answer "just do X" in these situations even if I push it. Sometimes it gets a lot more precise if I give more context, like

- How to compose a mix? What parts of the tracks should I play and what should I skip?

- It depends on lots of factors <type of party, music genre, the crowd, etc>. There is no one size fits all answer.

- Ok, suppose I am playing an early morning set of melodic techno at a houseparty for friends. What would you suggest?

- Then your friends will probably be tired by the morning so it makes sense to insert more breakdowns, melodic techno is usually structured this way, hence blah blah blah.

Or you can ask for examples. I often struggle with distinguishing closely related EDM subgenres:

- Can you explain the difference between psybass and psybreaks?

- <a vague description>

- I still don't get it. Can you give me 5 iconic tracks in both?

- Sure thing, here is the list: <..>

Besides, there are open research problems in CS as well where there is no single solution. Then I learn the vocabulary and google recent papers on this subject.

I think fundamentally we have a lot more information available than a single person can ever handle. I find ChatGPT incredibly useful at navigating and summarizing this sea of information.

When learning a new thing it’s very good for compiling knowledge from lots of sources and laying it out in a good format. As long as it’s easy to verify when it screws up.

Marketing text. As an engineer I can write factual statements, but can't spice it up the way a marketer could. I can give GPT-4 some basic facts then have it generate multiple examples, then pick one.

Generating plan names.

Customer interview recruitment strategies.

Brainstorming a customer interview plan.

Mock customer interviews.

Analytics. This is more technical but it's created SQL queries to understand all kinds of metrics. Also helping to understand the results.

I'm aware of the hallucination problem so I'll usually check other sources for anything important.

>Kind of like an AI CEO advising me on what to do next.

Or like someone who never was a CEO, but has absorbed all kinds of articles, listicles, and books with business advice (most of them written by hustle hacks, non-CEO content creators, or failed small-time enterpreneurs) and serves advice based on that...

I’m not doubting that you have found it helpful. I just wish people would be more judicious than making up numbers like “10x”. Everything can’t be “10x”.

In my case, if you are talking about startups specifically, its true. I've been an engineer now for 15 years. About 5 years ago, long before LLMs became a thing, I got to the point where I stopped using stack overflow and google for most things, only using them for the hardest, most complex questions. Thats how I use LLMs nowdays for technical things, which is not often. Business however is new to me.

As a solo founder, i've got to be the ceo, marketer, product manager, customer service rep, tech support person and a bunch of other things i've not previously done. GPT-4 is great at guiding me on these things. In a typical 2 person CEO/CTO founded early stage startup, the CEO might be doing these non technical things until the business gets enough funding to hire their first employees.

Please disclose your startup name so that we can stay far far away from whatever havoc you unleash.

Wow

This is extremely rude and downright uncomfortable to read on a forum where people are supposed to be curious.

Who tf do you think you are?

“Let me just use a glorified autocomplete instead of learning a subject domain or craft” seems like the opposite of hacker spirit and curiosity.

"Let's learn this new modern tool and how I can utilise it to fulfill my goals, while avoiding its shortcomings and flaws" sounds like the epitome of hacker spirit and curiosity. "Let's discredit this new technology as glorified autocomplete, and pretend it doesn't exist" is the opposite.

If it implies not having to learn anything new (or learning as little as possible) in order to get a result fast, then the correct term is “script-kiddie spirit” not “hacker spirit”.

Do you feel the same about using search engines to learn literally anything? What about youtube learning channels? What about the documentation? You know, hackers from the past could write amazing things in assembly using only an outdated book on a somewhat related subject (if any at all). We have invented internet to share knowledge as quickly as possible. ChatGPT is the next step.

What does a true Scotsman, sorry, hacker have to do? Go to a remote place and rediscover the entire human knowledge for 20 years?

The key word is “learn”!

I would love to understand this better.

Why do you think it is anti-curiosity to use LLMs?

The original commenter did not say they used it for producing content, so and auto complete is not really precise, is it?

You could say that it is an advanced, stochastic knowledge-retrieval system. Does the workings of such merit your curiosity? Does the workings of compression algorithms merit your curiosity?

Or are you merely and ignorant person already left behind due to your own inability to be curious?

>Why do you think it is anti-curiosity to use LLMs?

"Don't need to study and trial/error to learn about business, I can get business advice in a box".

It's Stack Overflow copy-pasting MkII.

This reads like a comment from a person who has not tried to integrate LLMs into various workflows.

Have you tried making and LLM draft a study plan for you? Kickstarting learning of a new area?

Have you tried making an LLM correct your "homework" for your studies? When you try to learn a new language?

Curiosity is about not being categorical - "It's Stack Overflow copy-pasting MkII." is extremely categorical and thus, anti-curiosity.

>This reads like a comment from a person who has not tried to integrate LLMs into various workflows

This reads like a comment from a person who makes the discussion about the other person.

>Curiosity is about not being categorical - "It's Stack Overflow copy-pasting MkII." is extremely categorical and thus, anti-curiosity.

You could say the exact same thing about someone criticizing regular Stack Overflow copy-pasting as lacking curiocity and hacking spirit. "Oh, that critique is so extremely categorical".

Sorry, "that's categorical" is not the trump card you think it is.

The root for this is a comment that urges to boycott some persons startup because they express that they were thankful for the advice they get from LLMs (a PhD in physics, that is).

Yes, I find that commentary extremely inappropriate. And yes, I am defending my stance in the sub-threads.

But tell me: you say that my critique of a reductionist categorical view on the use of LLMs (it is merely stack overflow...) is catagorical itself? Can you please elaborate on that.

>The root for this is a comment that urges to boycott some persons startup because

I didn't write it so don't care about the "root for this".

I quoted what I was commenting upon: "Why do you think it is anti-curiosity to use LLMs?".

>Yes, I find that commentary extremely inappropriate.

Feel free to do so. Has nothing to do with my comments and it's not a responce to my arguments. It's sidelining them and trying to appeal to outrage and emotion by getting back to what a different person said.

You do enter into a context when you decide responding within a thread - let's see if we can forget this.

Your sentiment is that this is stack overflow c/p style business development. I understand this as: You receive instructions from a third party entity and carry them out verbatim. An because that to use a LLM that is anti-curiosity.

Let's delve into that, shall we?

1. Following instructions verbatim is as old as instructions. You could apply the same sentiment to mentorship, books, and well, Q/A systems. Just to inspire some reflection: Do you think good chefs ever use recipes or is that only bad chefs? It is quite well established that following instructions is key to learning.

2. You say that it is equivalent to follow instructions from an LLM to following instructions from SO - I think this misses a key feature of LLMs: The ability to index them. You can tell them they are care about business development in a narrow sense, to what it is index all future responses. This is realtime interactive and does probe behavior like changing what you index your instructions for, which indeed is the core of curiosity.

Based on this I hold on to my previous statement as a response yo your comment:

> Curiosity is about not being categorical - "It's Stack Overflow copy-pasting MkII." is extremely categorical and thus, anti-curiosity.

I do find that your line of thought is narrow and dismissed the elements to be curious about regarding LLMs.

>You do enter into a context when you decide responding within a thread - let's see if we can forget this.

Oh, please

In my comment I objected to “hacker spirit” and “curiosity” being misappropriated to mean what they don’t (in a way, something opposite); if I wanted to support the call for boycott, I would have written so.

Treating a comment here as endorsing or refuting the entire thread leading up to it is not going to help a nuanced discussion.

It sounded like the author is not trying to learn, say, how to run a business, but instead just proxies output of an unthinking tool. After the former, you come out with new knowledge and skill. After the latter, you come out none the wiser.

> Does the workings of such merit your curiosity?

Not really, if the workings of such are well-understood (like in case of LLMs).

> Or are you merely and ignorant person already left behind

You think you are mounting a personal attack that would make me feel humiliated, but you need to check who would be ignorant after what I outlined in the beginning of my comment.

> ... but instead just proxies output of an unthinking tool. After the former, you come out with new knowledge and skill. After the latter, you come out none the wiser.

This reads like "bla bla bla" without any support. Let's try to dissect what you are saying

1. You are saying that LLMs are "unthinking tools" what is that? It does appear to be a made up word. What does it mean?

2. You set up a dichotomy between using this "unthinking tool" and running a business. Can you please point me to the passage where the commenter is saying that they are only doing one or the other, and not both?

3. You say that you become non the wiser by using this "unthinking tool". Would you extend that to all similar activities? Such as reading a book, reason blog articles, receiving lectures etc?

> but you need to check who would be ignorant after what I outlined in the beginning of my comment.

After you establishing false dichotomies and using non-existing words? You demonstrate neither being able to reason or use knowledge. The hallucinations from an LLM is strictly more useful that the hallucinations from you - At least they can be corrected.

HN has plenty of bad takes, and this is up there with the worst. OP is obviously learning from the LLM. What is with this knee-jerk reflex to complain about AI regardless of whether your complaint makes any sense?

It sounds like OP consults an LLM on each next step. “Apply pressure and move the brush to the right” is not how you learn to draw, “press the 44th key counting from the left” is not how you learn to play piano. Not to say LLM can’t be used to learn things, like any tool, with its own strengths and weaknesses.

There’s a proverb (I can’t remember where I read it now) that goes something like, “what comes in through the front gate isn’t the family jewels.” Seems appropriate here. There are no short cuts to building insight and and intuition.

Indeed, but a bigger gates takes more effecient machinary for the jewel production in the back.

You could use the same proverb to dismiss the effects of Gutenbergs printing press.

And also, now when you respond again: can you elaborate on why you feel it is necessary to urge people to boycott a startup, because they use a technique that you currently are not familiar with? I see that you have earned your PhD and I would not expect this kind of behavior from a person who has credentials like you.

I’ve dabbled with using LLMs, but in my experience they’re not even as good as using an annotated bibliography to help guide your study of a new domain. A bibliography, even a mediocre one, will give you an entry point into a literature, which can greatly speed up your exploration of the space. You still have to put in the work to synthesize and understand the literature and how it applies to your problem. Until LLMs can stop fabricating sources and concepts, they are not even as useful to research as an annotated bibliography. I suppose if no such bibliography exists, an LLM is better than nothing, but it must be used with extreme caution, and I’m not really sure how much time they would save than simply spending a week or two working through the literature for a domain using library search engines.

That's cool, seems like you have something that works well for you. Something which is also based on several hundreds of years of refinement.

And yet you are still extremely dismissive and rude around people investigating methods where foundational work is roughly 40 years old.

Humbleness and perspective can nobody blame on you, though - maybe an llm could help you on that?

What sort of questions do you ask it?