News for hackers. Stuff that matters.
Re:Deep Research by hi_hi on Sunday February 02, 2025 @07:47PM (#42913579)
This is terrifying. Even though they acknowledge the issues with hallucinations/errors, that is going to be completely overlooked by everyone using this, and then injecting the outputs into their own powerpoints.

Management Consulting was bad enough before the ability to mass produce these graphs and stats on a whim. At least there was some understanding behind the scenes of where the numbers came from, and sources would/could be provided.

The more powerful these tools become, the more prevelant this effect of seepage will become.

Re:Deep Research by autoconfig on Sunday February 02, 2025 @08:01PM (#42913709)
Either you care about being correct or you don't. If you don't care then it doesn't matter whether you made it up or the AI did. If you care then you'll fact check before publishing. I don't see why this changes.
Re:Deep Research by spaceywilly on Sunday February 02, 2025 @08:10PM (#42913768)
I think a lot about how differentiating facts and quality content is like differentiating signal from noise in electronics. The signal to noise ratio on many online platforms was already quite low. Tools like this will absolutely add more noise, and arguably the nature of the tools themselves make it harder to separate the noise.

I think this is a real problem for these AI tools. If you can’t separate the signal from the noise, it doesn’t provide any real value, like an out of range FM radio station.

Re:Deep Research by WOTERMEON on Sunday February 02, 2025 @08:37PM (#42913967)
Not only that: by publishing noise, you’re lowering the signal/noise ratio.
Re:Deep Research by layer8 on Sunday February 02, 2025 @08:19PM (#42913843)
People are much less scrupulous using LLM output than making up stuff themselves, because then they can blame the LLM.
Re:Deep Research by hi_hi on Sunday February 02, 2025 @08:19PM (#42913844)
Because maybe you want to, but you have a boss breathing down your neck and KPIs to meet and you haven't slept properly in days and just need a win, so you get the AI to put together some impressive looking graphs and stats that will look impressive in that client showcase thats due in a few hours.

Things aren't quite so black and white in reality.

Re:Deep Research by dauhak on Sunday February 02, 2025 @09:07PM (#42914162)
I mean those same conditions already just lead the human to cutting corners and making stuff up themselves. You're describing the problem where bad incentives/conditions lead to sloppy work, that happens with or without AI

Catching errors/validating work is obviously a different process when they're coming from an AI vs a human, but I don't see how it's fundamentally that different here. If the outputs are heavily cited then that might go someway into being able to more easily catch and correct slip-ups

Re:Deep Research by hi_hi on Sunday February 02, 2025 @09:33PM (#42914305)
Yep, I agree with this to some extent, but I think the difference in the future is all that stress will be bypassed and people will reach for the AI from the start.

Previously there was alot of stress/pressure which might or might not have led to sloppy work (some consultants are of a high quality). With this, there will be no stress which will (always?) lead to sloppy work. Perhaps there's an argument for the high quality consultants using the tools to produce accurate and high quality work. There will obviously be a sliding scale here. Time will tell.

I'd wager the end result will be sloppy work, at scale :-)

Re:Deep Research by tikhonj on Monday February 03, 2025 @12:30AM (#42915282)
Making it easier and cheaper to cut corners and make stuff up will result in more cut corners and more made up stuff. That's not good.

Same problem I have with code models, honestly. We already have way too much boilerplate and bad code; machines to generate more boilerplate and bad code aren't going to help.

Re:Deep Research by mquander on Monday February 03, 2025 @12:36AM (#42915305)
The technology also makes it easier and cheaper to make good things, so the direction of the outcome isn't guaranteed.
Re:Deep Research by sbarre on Sunday February 02, 2025 @08:55PM (#42914080)
How hard it is to produce credible-looking bullshit makes a really big difference in these scenarios.

Consultants aren't the ones doing the fact-checking, that falls to the client, who ironically tend to assume the consultants did it.

Re:Deep Research by azinman2 on Sunday February 02, 2025 @09:00PM (#42914117)
When things are easy, you’re going to take the easy path even if it means quality goes down. It’s about trade offs. If you had to do it yourself, perhaps quality would have been higher because you had no other choice.

Lots of kids don’t want to do homework. That said, previously many would because there wasn’t another choice. But now they can just ask ChatGPT for the answers they’ll write that down verbatim with zero learning taking place.

Caring isn’t a binary thing or works in isolation.

Re:Deep Research by simonw on Sunday February 02, 2025 @10:46PM (#42914715)
"Lots of kids don’t want to do homework"

Sure, but if you're a professional you have to care about your reputation. Presenting hallucinated cases from ChatGPT didn't go very well for that lawyer: https://www.nytimes.com/2023/05/27/nyregion/avianca-airline-...

Re:Deep Research by PeterStuer on Monday February 03, 2025 @01:36AM (#42915577)
That's a lawyer in an adverserial situation. Business consultants tell their clients what they want to believe, the facts be dammed.
Re:Deep Research by rsanek on Monday February 03, 2025 @02:13AM (#42915751)
it sounds like ai doesn't really change that situation
Re:Deep Research by asimpletune on Monday February 03, 2025 @02:57AM (#42915956)
But the point is it does if you count making it worse changing the situation.
Re:Deep Research by financypants on Monday February 03, 2025 @12:16AM (#42915214)
what about tests?
Re:Deep Research by jstummbillig on Monday February 03, 2025 @04:04AM (#42916335)
I don't think it follows that taken an easier path would mean quality goes down.
Re:Deep Research by ADeerAppeared on Sunday February 02, 2025 @09:22PM (#42914243)
> If you care then you'll fact check before publishing.

Doing a proper fact check is as much work as doing the entire research by hand, and therefore, this system is useless to anyone who cares about the result being correct.

> I don't see why this changes.

And because of the above this system should not exist.

Re:Deep Research by RainyDayTmrw on Sunday February 02, 2025 @11:53PM (#42915119)
It's possible that you care, but the person next to you doesn't, and external pressures force you to keep up with the person who's willing to shovel AI slop. Most of us don't have a complete luxury of the moral high ground at our jobs.
Re:Deep Research by doomroot on Monday February 03, 2025 @12:00AM (#42915145)
It looks like the moral high just came more in demand.
Re:Deep Research by navigate8310 on Monday February 03, 2025 @03:50AM (#42916253)
It's the high reps fault then of not caring about quality. Either you assimilate in that low quality lower management using AI slop or change job.
Re:Deep Research by n4r9 on Monday February 03, 2025 @02:30AM (#42915840)
It's a bit like saying "my kids are going to hit themselves anyway, so it doesn't matter if I give them foam rods or metal rods".
Re:Deep Research by ctoth on Monday February 03, 2025 @10:54AM (#42919423)
Maybe this would make sense if you saw the whole world as "kids" that you had to protect. As an adult who lives in an adult world, I would like adults to have access to metal tools and not just foam ones.
Re:Deep Research by n4r9 on Monday February 03, 2025 @11:11AM (#42919675)
I guess I can replace "kid" with "toddler" and add "unsupervised" at the end.
Re:Deep Research by mlsu on Monday February 03, 2025 @12:41PM (#42920726)
If 20% of people don't care about being correct, the rest of everyone can deal with that. If 80% of people don't care about being correct, the rest of us will not be able to deal with that.

Same thing as misinformation. A sufficient quantitative difference becomes a qualitative difference at some point.

Re:Deep Research by michael_swift on Tuesday February 04, 2025 @01:39PM (#42936702)
don't you think the problem of checking for correctness then becomes more insidious then? we now can generate hundreds of reports that look very professional on the surface. the usual things that would tip you off that this person was careless aren't there -- typos, poor sentence construction, missing references. just more noise to pick through for signal
Re:Deep Research by scarab92 on Sunday February 02, 2025 @08:02PM (#42913713)
Think of it like a vaccine.

The majority of consultant reports are complete rubbish. Low signal-to-noise, quantity-over-quality, low accuracy slop.

LLMs are innoculating people to this kind of low information value content.

People who produce LLM quality output, are now being accused of using LLMs, and can no longer pretend to be adding value.

The result of this is going to be higher quality expectations from consultants and a shaking out of people who produce word vommit rather than accurate, insightful, contextually relevent information.

Re:Deep Research by layer8 on Sunday February 02, 2025 @08:24PM (#42913886)
This has been downvoted, but I think there’s actually a chance it might become true (until AGI comes along at least).
Re:Deep Research by DrSiemer on Monday February 03, 2025 @02:47AM (#42915909)
Exactly what will happen with art. The tolerance for low quality output will decrease.
Re:Deep Research by randcraw on Monday February 03, 2025 @12:41PM (#42920728)
I don't think so. Instead of SEO, I think we'll soon see 'LLMO' dominating such uses, where LLM summaries are reshaped by vendors and etailers to misrepresent facts in ways that favor them over others.

I suspect this can be done simply by poisoning a query with supplemental suggestions of sources to use in a RAG, many of which don't even have to be publicly available but are made accessible to the LLM (perhaps by submitting hidden URLs that mislead the summary along with the query).

But even after such a practice is uncovered and roundly maligned, that won't stop the infinite supply of net con men from continuing to inject their poisons into the background that drives deep research, so long as the LLM maker doesn't actively oppose this practice actively and publicly -- which none of them have been willing to do with any other LLM operational details so far.

In fact, I predict that if a LLM summary like DR's does NOT soon provide references to the sources of the facts it relies on, in no time users will disregard such summaries to be yet more uselessly unreliable pfaff from yet another net disreputable -- as we do with search engine summaries now.

Re:Deep Research by tmnvdb on Sunday February 02, 2025 @07:54PM (#42913642)
> At least there was some understanding behind the scenes of where the numbers came from, and sources would/could be provided.

Oh Sweet summer child.

Re:Deep Research by opdahl on Monday February 03, 2025 @10:03AM (#42918843)
Hi tmnvdb, since you seem to love these super smart LLMs I thought it would be fun to have openais o3-mini-high analyze your recent comments in contrast to the Hacker News Comment Guidelines. Here is the output it gave me, hope it helps you:

------

Hey, I've noticed a few things in your style that are both strengths and opportunities for improvement:

Strengths:

- You clearly have deep knowledge and back up your points with solid data and examples.

- Your confidence and detailed analysis make your arguments compelling.

Opportunities:

- At times, your tone can feel a bit combative, which might shut down conversation.

- Focusing on critiquing ideas rather than questioning someone's honesty can help keep the discussion constructive.

- A clearer structure in longer posts could make your points even more accessible.

Overall, your passion and expertise shine through—tweaking the tone a bit might help foster even more productive debates.

------

Just reply here if you want the full 500+ words analysis that goes into more detail.

Re:Deep Research by _bin_ on Sunday February 02, 2025 @11:16PM (#42914919)
let's be real for a sec, i've done consulting and have a lot of friends who still do. three times in four, your mckinsey report isn't super well-founded in reality and involves a lot of guesstimation.
Re:Deep Research by n144q on Monday February 03, 2025 @12:23AM (#42915243)
I think that ship has sailed many years ago since Facebook allowed false information to spread freely on their site (if not earlier).
Re:Deep Research by anthonyshort on Monday February 03, 2025 @01:34AM (#42915568)
Then the hallucinated research is published in an article which is then cited by other AI research, continuing the push the false information until it’s hard to know where the lie started.