How do you, in general, fact check a chain of reasoning?

I can’t tell a search engine to summarize text for a technical audience and then another summary for a non technical audience.

I recently came into the middle of a cloud consulting project where a lot of artifacts, transcripts of discovery sessions, requirement docs, etc had already been created.

I asked NotebookLM all of the questions I would have asked a customer at the beginning of a project.

What it couldn’t answer, I then went back and asked the customer.

I was even able to get it to create a project plan with work streams and epics. Yes it wouldn’t have been effective if I didn’t already know project management, AWS and two decades+ of development experience.

Despite what people think, LLMs can also do a pretty good job at coding when well trained on the APIs. Fortunately, ChatGPT is well trained on the AWS CLI, SDKs in various languages and you can ask it to verify the SDK functions on the web.

I’ve been deep into AWS based development since LLMs have been a thing. My opinion may change if I get back into more traditional development

No, but, as amazing as that is, don't put too much trust in those summaries!

It's not summarizing based on grokking the key points of the text, but rather based on text vs summary examples found in the training set. The summary may pass a surface level comparison to the source material, while failing to capture/emphasize the key points that would come from having actually understood it.

Just like I’m not randomly depending on it to do an Amazon style PRFAQ (I was indoctrinated as an Amazon employee for 3.5 years), create a project plan, etc, without being a subject matter expert in the areas. It’s a tool for an experienced writer, halfway decent project manager, AWS cloud application architect and developer.

There is an entire “reproducibility crisis” with research.

Try training an LLM.