News for hackers. Stuff that matters.
Re:'Positive review only': Researchers hide AI prompts in pa... by grishka on Saturday July 05, 2025 @05:20PM (#44475641)
Well, that's the thing — if you understand the technology you're working with and know how to verify the result, chances are, completing the same task with AI would take you longer than without it. So the whole appeal of AI seems to be to let it do things without much oversight.

The common failure mode of AI is also concerning. If you ask it to do something that can't be done trivially or at all, or wasn't present enough in the learning dataset, it often wouldn't tell you it doesn't know how to do it. Instead, it'll make shit up with utmost confidence.

Just yesterday I stumbled upon this article that closely matches my opinion: https://eev.ee/blog/2025/07/03/the-rise-of-whatever/

Re:'Positive review only': Researchers hide AI prompts in pa... by serbuvlad on Saturday July 05, 2025 @05:47PM (#44475831)
But that's exactly the thing. I DON'T understand the technology without AI.. I know stuff about Linux, but I knew NOTHING about Ansible, FreeIPA etc. So I guess you could say I understand the problem space not the solution space?? Either way, it would have taken us many months to do what it did take us a few weeks to with AI.

> So the whole appeal of AI seems to be to let it do things without much oversight.

No?? The whole appeal of AI for me is doing things I know how I want to look at the end but I don't know how to get there.

> The common failure mode of AI is also concerning. If you ask it to do something that can't be done trivially or at all, or wasn't present enough in the learning dataset, it wouldn't tell you it doesn't know how to do it. Instead, it'll make shit up with utmost confidence.

I also feel like a lot of people made a lot of conclusions against GPT-3.5 that simply aren't true anymore.

Usually o3 and even 4o and probably most modern models rely a lot more on search results then on their training datasets. I usually even see "I know how to do this but I need to check the documentation for up to date information in case anything changed" in the chain of thought for trivial queries.

But yeah, sometimes you get the old failure mode: stuff that doesn't work. And then you try it and it fails. And you tell it it fails and how. And it either fixes it (90%+ of cases, at least with something powerful like o3), or it starts arguing with you in a nonsensical manner. If the latter, you burn the chat and start a new one, building better context, or just do a manual approach like before.

So the failure mode doesn't mean you can't identify failure. The failure mode means you can't trust it's unchecked output. Ok. So? It's not a finite state machine, it's a statistical inference machine trained on the data that currently exists. It doesn't enter a faliure state. Neither does a PID regulator when the parameters of the physical model change and no one recalibrates it. It starts outputting garbage and overshooting like crazy etc.

But both PID regulators and LLMs are hella useful if you have what to use them for.

Re:'Positive review only': Researchers hide AI prompts in pa... by soraminazuki on Saturday July 05, 2025 @10:12PM (#44477241)
> I know stuff about Linux, but I knew NOTHING about Ansible, FreeIPA etc.

Then you absolutely shouldn't be touching Ansible or FreeIPA in production until you've developed enough understanding of the basics and can look up reliable sources for the nitty gritty details. FreeIPA is security critical software for heaven's sake. "Let's make up for zero understanding with AI" is a totally unacceptable approach.

Re:'Positive review only': Researchers hide AI prompts in pa... by serbuvlad on Sunday July 06, 2025 @01:39AM (#44478124)
Missed the part where I said that:

a) I develop the understanding with AI (I would never use something I don't understand at all),

b) I test before pushing to prod and

c) This replaces a bunch of shoddy shell scripts so even if there are hiccups, there were a lot more hiccups before?

Re:'Positive review only': Researchers hide AI prompts in pa... by gridspy on Sunday July 06, 2025 @10:03PM (#44486047)
Often the problem best solved by AI isn't "How do I use Anisble to do X" But simply knowing that you should be using Ansible at all. Or which features to use. Then you can be laser-focused on learning exactly the right part of Ansible.

It also helps you to move away from familiar but hacky solutions towards much more standard and robust ones - the AI doesn't approach the problem with your many years of battle scars and baggage but instead sees your problem space with fresh eyes and recent data.

Re:'Positive review only': Researchers hide AI prompts in pa... by grishka on Sunday July 06, 2025 @10:31PM (#44486198)
But why use AI for that in the first place? You can just, uh, google things. Like "software that does X". You would get a better result from forums and blogs and such. When doing research, AI really feels like an unnecessary middleman when you know how to do research on the internet. Which I would argue is a basic skill in the modern world.
Re:'Positive review only': Researchers hide AI prompts in pa... by reseasonable on Monday July 07, 2025 @10:56AM (#44491052)
So your point went from: they shouldn’t have done what they did because it likely took much longer than necessary by using AI - to: they shouldn’t have done what they did because they didn’t take long enough learning it without AI to do it. I think you moved the goalpost to the opposite side of the field.

In this thread you chastised them for “using it in production” when their use case was simply a lab for teaching. And you suggest that googling for days and reading countless blog posts and forums is impervious to wrong or dated information, which is quite a position to take.

Their use case was a perfect fit for a LLM, and I suspect it saved them weeks of effort in the process, with likely fewer security flaws as a result. But keep up that anti-LLM advocacy, I’m sure you are very close to achieving the goal of society abandoning the technology.

Re:'Positive review only': Researchers hide AI prompts in pa... by gridspy on Monday July 07, 2025 @06:05PM (#44495116)
Keyword based search (Google) requires the user to know which keywords to enter. It's very bad at finding things related but with different words to what the user is trying to search. Sometimes half the challenge is figuring out what the experts call the task you are trying to achieve.

Set up some desktops with code -> Declarative configuration management -> Ansible

This task seems easy to an expert. As someone who's been a professional learner (as a programmer) for 25 years, it's far far easier with ChatGPT to get a summary of a field and a starting point than it ever was with just Google and some random blogs bouncing around the internet. Now with modern o3 (Chat GPT Pro) models you can go from a vague question to a summary of suitable solutions and some code samples in one prompt.

Now obviously AI is not able to create complete solutions alone. It's just another tool. Sometimes ChatGPT makes glaring mistakes in how it approaches a problem. So (for now) it is still a collaboration requiring smart humans in the loop.