Re:Everything around LLMs is still magical and wishful thinking
by troupo on Saturday July 05, 2025 @01:00AM
(#44470186)
> 12 months ago, if I fed a list of ~800 poems with about ~250k tokens to an LLM and asked it to summarize this huge collection, they would be completely blind to some poems and were prone to hallucinating not simply verses but full-blown poems.
for the past week claude code has been routinely ignoring CLAUDE.md and every single instruction in it. I have to manually prompt it every time.
As I was vibe coding the notes MCP mentioned in the article [1] I was also testing it with claude. At one point it just forgot that MCPs exist. It was literally this:
> add note to mcp
Calling mcp:add_note_to_project
> add note to mcp
Running find mcp.ex
... Interrupted by user ...
> add note to mcp
Running <convoluted code generation command with mcp in it>
We have no objective way of measuring performance and behavior of LLMs
for the past week claude code has been routinely ignoring CLAUDE.md and every single instruction in it. I have to manually prompt it every time.
As I was vibe coding the notes MCP mentioned in the article [1] I was also testing it with claude. At one point it just forgot that MCPs exist. It was literally this:
We have no objective way of measuring performance and behavior of LLMs[1] https://github.com/dmitriid/mcp_notes