News for hackers. Stuff that matters.
Re:Amazon Introduces Q, an A.I. Chatbot for Companies by rurp on Tuesday November 28, 2023 @01:45PM (#38449524)
I just noticed Q in the AWS docs and tried a few test questions, and was not impressed. It refused to answer or misunderstood some basic questions. Eventually I got it to answer how a few short SKs would be ordered in dynamo and the answer it gave was incorrect.

Technical documentation is probably one of the worst usecases for GenAI, I'm not sure why so many companies are rushing to add it.

Re:Amazon Introduces Q, an A.I. Chatbot for Companies by acdha on Tuesday November 28, 2023 @03:10PM (#38450574)
I think docs are tempting because it’s a mountain of content, customers always ask about changes, and the senior managers tend not to respect documentation team and view them as pure cost.
Re:Amazon Introduces Q, an A.I. Chatbot for Companies by freshpots on Tuesday November 28, 2023 @02:11PM (#38449903)
"Technical documentation is probably one of the worst usecases for GenAI, I'm not sure why so many companies are rushing to add it."

I am one of those people who think that it would help people summarize it, get better compliance with specs, etc.

However, I am limited in my knowledge when it comes to GenAI.

Why do you think it is one of the worst use cases?

Re:Amazon Introduces Q, an A.I. Chatbot for Companies by rurp on Wednesday November 29, 2023 @01:18PM (#38462995)
Because being precisely correct is especially important in technical documentation. GenAI is great at producing loads of content that looks sort of right, but terrible at logical correctness and, to a lesser extent, brevity.

In my original example I asked Q about sorting in DynamoDB. The answer it gave was categorically wrong! That's worse than useless, it's actively misleading. If it can't get a simple example correct I have no faith that it will be reliable for more comicated real world questions, but those mistakes will be harder to catch.

Re:Amazon Introduces Q, an A.I. Chatbot for Companies by lazystar on Wednesday November 29, 2023 @07:22PM (#38467465)
as another anecdote, Q advised me to update a security group with a rule to allow ingress traffic on specific ports from a Lambda function execution role. It's jumbling up documentation from two completely different security tools; it's only a matter of time before Q causes a prod incident somewhere.
Re:Amazon Introduces Q, an A.I. Chatbot for Companies by ChicagoBoy11 on Tuesday November 28, 2023 @02:31PM (#38450111)
I mean, surely the answer has to be: "because it's a pain point and there's a market for it", no? In the sense that, I think your skepticism is (rightly) warranted, but only because of the outcome that you've seen so far has produced unsatisfying results. In a universe where this approach does yield consistently correct and succinct answers, having an AI read a large body of technical documentation and be able to serve you the exact answer that you need from it does seem like a solution with lots of takers!
Re:Amazon Introduces Q, an A.I. Chatbot for Companies by danielmarkbruce on Tuesday November 28, 2023 @02:54PM (#38450367)
> Technical documentation is probably one of the worst usecases for GenAI, I'm not sure why so many companies are rushing to add it.

Why?

Re:Amazon Introduces Q, an A.I. Chatbot for Companies by chrisfowles on Wednesday November 29, 2023 @07:08AM (#38458421)
Because it's unique to the use case, complex, and nuanced. It has to be exactly correct, and it has potentially serious ramifications if the answers are incorrect, potentially with legal ramifications that I'm keenly waiting to see the first cases of.
Re:Amazon Introduces Q, an A.I. Chatbot for Companies by PeterisP on Wednesday November 29, 2023 @09:17AM (#38459574)
Quite the contrary, just as for many products already it was better to search stackoverflow than the technical documentation, already last year one of the (few) use cases where I found ChatGPT really useful was asking for examples for various basic things that kind of are decribed in the technical documentation (and also on stackoverlow, and on a bajillion blogs) but the usability of that documentation (and intentional choices of its design, what to include and what not) sucks so much that it's far, far more effective to just ask a chatbot than read the original docs.