If a topic is well represented in those places, then you will get your answer quicker and it can be to some extent shaped to your use case.

If the topic is not well represented there, then you will get circular nonsense.

You can say “obviously, that’s the training data”, and that’s true, and I do find it obvious personally, but the reaction to LLMs as some kind of second coming does not align with this reality.

If I have a question about a first order language or framework feature or pattern, it works great. If I have a question about a second order problem, like an interaction between language or framework features, or a logical inconsistency in feature behavior, then it usually has no idea what’s going on, unless it turns out to be a really common problem such as something that would come up when working through a tutorial.

For code completion, I’ve just turned it off. It saves time on boilerplate typing for sure, but the actual content pieces are so consistently wrong that on balance I find it distracting.

Maybe I have a weird programming style that doesn’t mesh well with the broader code training corpus, not sure. Or maybe a lot of people spend more time in the part of problem-space that intersects with tutorial-space? I am not very junior these days.

That being said I definitely do use LLMs to engage with tutorial type content. For that it is useful. And outside of software it is quite a bit better for interfacing with Wikipedia type content. Except for the part where it lies to your face. But it will get better! Extrapolating never hurt anyone.

It's pretty good at writing screens in broad strokes. You will have to fill in some details.

The exact details of correctly threading data through; or prop drilling vs alternatives; the rules around wrapping screens to use them in React Navigation? It's terrible at them.

So you're using it like Wikipedia? I find when learning something new (and non coding related) YouTube is infinitely better than an LLM. But then I prefer visual demonstration to tutorial or verbal explanation.

Did you puzzle about this sentence specifically? Imagine you don't know jack about flying helicopters, then Tank uploads the Helicopter Pilot Program (TM) directly to your brain; it would feel like magic.

Conversely, if you know a lot about helicopters, just not enough to fly a B-212, and the program includes instructions like "Press (Y) and Left Stick to hover", you'd know it's confabulating real world piloting with videogames.

That's the same with LLMs, you need to know a lot of the field to ask the right questions, and recognize/correct slop or confabulation, otherwise they seem much more powerful and smart than they really are.

I think that its because often the libraries you use are niche or have a a few similar versions, the LLM really commonly hallucinated solutions and would continually suggest that library X did have that capability. I think because often in hardware projects you often hit a point where you can't do something or you need to modify a library, but the LLM tries to be "helpful" and it makes up a solution.

And then you’ll paste in the error, and they’ll just say “ok I see the problem” and output the exact same broken code lol.

I’m guessing the problem is lack of training data. Most TS codebases are mostly just JS with a few types and zod schemas. All of the neat generic stuff happens in libraries or a few utilities

The number of projects I’ve done where my notes are the difference between hours of relearning The Way or instant success. Google doesn’t work as some niche issue is blocking the path.

ESP32, Arduino, Home Assistant And various media server things.

Public Arduino, RPi, Pico communities are basically peak cargo cult, with the blind leading the blind through things they don't understand. The noise is vastly louder than the signal.

There's a basically giant chasm between expereinced or professional embedded developers that mostly have no need to ever touch those things or visit their forums, and the confused hobbyists on those forums randomly slapping together code until something sorta works while trying to share their discoveries.

Presumably, those communities and their internal knowledge will mature eventually, but it's taking a long long time and it's still an absolute mess.

If you're genuinely interested in embedded development and IoT stuff, and are willing to put in the time to learn, put those platforms away and challenge yourself to at least learn how to directly work with production-track SoC'a from Nordic or ESP or whatever. And buy some books or take some courses instead of relying on forums or LLM's. You'll find yourself rewarded for the effort.

It won't because the RPi are all undocumented, closed-source toys.

It would be an interesting experiment to see which chips an LLM is better at helping out with: RPi's with its hallucinatory ecosystem or something like the BeagleY-AI which has thousands of pages of actual TI documentation for its chips.

It would be really nice if the LLMs could cover for this and circumvent where RPi's keep getting used because they were dumped under cost to bootstrap a network effect.

I'm not sure they will. There's a kind of evaporative cooling effect where once you get to a certain level of understanding you switch around your tools enough that there's not much point interacting with the community anymore.

Eventually I read some actual documentation and realised it was just spouting very plausible sounding nonsense - and confident at it!

The same thing happened a year or so ago when I tried to get a much older ChatGPT to help me with with USB protocol problems in some microcontroller code. It just hallucinated APIs and protocol features that didn’t actually exist. I really expected more by now - but I now suspect it’ll just never be good at niche tasks (and these two things are not particularly niche compared to some).

For the best of both worlds make the LLM first 'read' the documentation, and then ask for help. Make a huge difference in the quality and relevance of the answers you get.

> I think the key to being successful here is to realize that you're still at the wheel as an engineer. The llm is there to rapidly synthesize the universe of information.

Bingo. OP is like someone who is complaining about the tools, when they should be working on their talent. I have a LOT of hobbies (circuits, woodworking, surfing, playing live music, cycling, photography) and there will always be people who buy the best gear and complain that the gear sucks. (NOTE: I"m not implying claude is "the best gear", but it's a big big help.)

I think the only problem with LLMs is synthesis of new knowledge is severely limited. They are great at explaining things others have explained, but suck hard at inventing new things. At least that's my experience with Claude: it's terrible as a "greenfield" dev.

My college professor was also willing to say "I don't know, ask me next class"

I just asked Claude a question I am pretty sure was not in its training data.

* https://i.imgur.com/XjvImeT.jpeg

https://www.quora.com/How-many-Humans-can-we-fit-on-the-Moon

Edit: couldn't resist, and dammit!!

Response: Ah, I see what you're doing! Since the Moon has no atmosphere, there’s technically no air to create any kind of airspeed velocity. So, the answer is... zero miles per hour. Unless, of course, you're asking about the speed of the horse itself! In that case, we’d just have to know how fast the astronaut can gallop without any atmosphere to slow them down.

But really, it’s all about the fun of imagining a moon-riding astronaut, isn’t it?

This is a perfect example of where not knowing the “domain” leads you astray. As far as I know “newborn width” is not something typically measured, so Claude is pulling something out of thin air.

Indeed you are showing that something not in the training data leads to failure.

Edit: it also doesn't account for the fact the moon is more or less a sphere, and not a flat plane.

This is a key differentiator that I see in humans over LLMs: knowing ones limits.

Granted, they are not (can't be) as rigorous as the tests your professor took, but new models are run through test suites before being released, too.

That being said, I saw my college professors making up things, too (mind you, they were all graduated from very good schools). One example I remember was our argument with a professor who argued that there is a theoretical limit for the coefficient of friction, and it is 1. That can potentially be categorised as a hallucination as it was completely made up and didn't make sense. Maybe it was in his training data (i.e. his own professors).

I agree with the "I don't know" part, though. This is something that LLMs are notoriously bad.

Also, you method for attesting your professors accuracy is inherently flawed. That little piece of paper on their wall doesn't correlate with how accurate they are; it doesn't mean zero, but it isn't foolproof. Hate to break it to you, but even heroes are fallible.

I don't use Claude, so maybe there's a huge gap in reliability between it and ChatGPT 4o. But with that disclaimer out of the way, I'm always fairly confused when people report experiences like these—IME, LLMs fall over miserably at even very simple pure math questions. Grammatical breakdowns of sentences (for a major language like Japanese) are also very hit-or-miss. I could see an LLM taking the place of, like, an undergrad TA, but even then only for very well-trod material in its training data.

(Or maybe I've just had better experiences with professors, making my standard for this comparison abnormally high :-P )

EDIT: Also, I figure this sort of thing must be highly dependent on which field you're trying to learn. But that decreases the utility of LLMs a lot for me, because it means I have to have enough existing experience in whatever I'm trying to learn about so that I can first probe whether I'm in safe territory or not.

They are language models, not calculators or logic languages like Prolog or proof languages like Coq. If you go in with that understanding, it makes a lot more sense as to their capabilities. I would understand the parent poster to mean that they are able to ask and rapidly synthesize information from what the LLM tells them, as a first start rather than necessarily being 100% correct on everything.

For as rich a culture the Japanese have, there's only about 1XX million speakers and the size of the text corpus really matters here, the couple billion of English speakers are also highly motivated to choose English over anything else because Lingua Franca has homefield advantage

To use LLM's efectively you have to work with knowledge of their weaknesses, Math is a good example, you'll get better results from Wolphram Alpha even for the simple things, which is expected

Broad reasoning and explanations tend to be better than overly specific topics, the more common a language, the better the response If a topic has a billion tutorials online, an LLM has a really high chance of figuring out first try

Be smart with the context you provide, the more you actively constrain an LLM, the more likely it is to work with you I have friends that just use it to feed class notes to generate questions and probe it for blindspots until they're satisfied, the improvements on their grade s make it seem like a good approach, but they know that just feeding responses to the LLM isn't trustworthy, so they do and then they also check by themselves, the extra time valuable by itself, if just to improve familiarity with the subject

YMMV. /shrugs/

But, once you've had AI help you solve some gnarly problems, it is hard not to be amazed.

And this is coming from a gal who thinks the idea of self-driving cars is the biggest waste of resources ever.

Sorry, maybe I should've been clearer in my response—I specifically disagree with the "college professor" comparison. That is to say, in the areas I've tried using them for, LLM's can't even help me solve simple problems, let alone gnarly ones. Which is why hearing about experiences like yours leaves me genuinely confused.

I do get your point about people disagreeing with modern AI for "political" reasons, but I think it's inaccurate to lump everyone into that bucket. I, for one, am not trying to make any broader political statements or anything—I just genuinely can't see how LLMs are as practically useful as other people claim, outside of specific use cases.

Since AI will capitulate and give you whatever you want.

You also have to learn how to ask without suggesting because it will take whatever you give it and agree.

I love LLMs. I agree with OP them expanding my hobby capacity as well. But I am constantly saying (in effect) “you sure…?” and tend to have a pretty good bs meter.

I’m still working to get my partner to that stage. They’re a little too happy to accept an answer without pushback or skepticism.

I think being ‘eager to accept an answer’ is the default mode of most people anyway. These tools are likely enabling faster disinformation consumption for the unaware.

Junior devs can get plenty of value out of them too, if they have discipline in how they use them - as a learning tool, not as a replacement for thinking about projects.

Senior devs can get SO much more power from these things, because they can lean on many years of experience to help them evaluate if the tool is producing useful results - and to help them prompt it in the most effective way possible.

A junior engineer might not have the conceptual knowledge or vocabulary to say things like "write tests for this using pytest, include a fixture that starts the development server once before running all of the tests against it".

Basically if they are good at utilizing junior developers and interns or apprentices they probably will do well with an LLM assistant.

Getting the LLM to pull out well-known names of concepts is for me the skill you can't get anywhere else. You can describe a way to complete a task and ask for what it's called and you'll be heading down arxiv links right away. Like yes the algorithm to find the closest in edit distance and length needle string in a haystack is called Needleman–Wunsch, of course Claude, everyone knows that.

* I put commercial there as a qualifier because there's some thought that in the future, very specifically-trained smaller models (open source) on particular technologies and corpuses (opt-in) might yield useful results without many of the ethical minefields we are currently dealing with.

On the other hand if you understand what needs to be done, and how to direct the work the productivity boost can be massive.

Claude 3.5 sonnet and O1 are awesome at code generation even with relatively complex tasks and they have a long enough context and attention windows that the code they produce even on relatively large projects can be consistent.

I also found a useful method of using LLMs to “summarize” code in an instructive manner which can be used for future prompts. For example summarizing a large base class that may be reused in multiple other classes can be more effective than having to overload a large part of your context window with a bunch do code.

That's complicated, but I wouldn't say the resulting software is complex. You gave an LLM a repetitive, translation-based job, and you got good results back. I can also believe that an LLM could write up a dopey SAAS in half the time it would take a human to do the same.

But having the right parameters only takes you so far. Once you click generate, you are trusting that the model has some familiarity with your problem and can guide you without needing assistance. Most people I've seen rely entirely on linting and runtime errors to debug AI code, not "solid fundamentals" that can fact-check a problem they needed ChatGPT to solve first place. And the "experience" required to iterate and deploy AI-generated code basically boils down to your copy-and-paste skills. I like my UNIX knowledge, but it's not a big enough gate to keep out ChatGPT Andy and his cohort of enthusiastic morons.

We're going to see thousands of AI-assisted success stories come out of this. But we already had those "pennies on the dollar" success stories from hiring underpaid workers out of India and Pakistan. AI will not solve the unsolved problems of our industry and in many ways it will exacerbate the preexisting issues.

Some of us are capable of wanting for things better than a coin-operated REST API. The kind of imagination used to put people on the moon, that now helps today's business leaders imagine more profitable ways to sell anime pornography on iPhone. (Don't worry, AI will disrupt that industry too.)

That was about 15 years ago, I no longer have the same enthusiasm you do.

No I am really not. You aren't either.

What is your main project ? Do you LLM that? (

I wager you're not a rust expert and should maybe reconsider using rust in your main project.

FWIW asking LLM whether you should use rust ~ asking it about the meaning of life. Important questions that need answers, but not right away! (A week or 2 tops)

If you need to synthesize the universe of information with LLM.. that is not the universe you want to live or play in