poytr1

Yes, I’m the author of this paper.

One correction: the paper is inaccurate on this point. If you look at the implementation(https://github.com/agentconnect-md/lsp-vs-grep-token-study/b...), the harness is actually based on Claude Code, with a system prompt and custom LSP tools registered.

Sorry for the confusion! :)

I had a similar intuition that we could definitely optimize token efficiency and performance for non-general tasks.

For example, I was handling a lot of infra operations, many of which follow fixed workflows like locating a specific log in certain time range, identify the issue, and apply the appropriate mitigation steps. Since these workflows are entirely command-line based, I added a hook to Claude Code that monitors the commands I repeatedly run, including those invoked by Claude Code itself, and compiles them into reusable scripts(with skills as entries). It can work well.

The takeaway is that we should spend more time reviewing our own work patterns and workflows.

Interesting tool, it seems that dedicated skills for these tools are important. Even explicit system-prompt instructions—such as “you must use tool X for code exploration”—often fail to behave as expected.

Another tricky part is that how to evaluate customized toolsets properly and ensure they continue to work perfectly as the underlying model/harness evolves.

I am thinking of where the initial training data came from. For example, Claude Code likely collected a substantial number of real-world coding trajectories through its CLI. However, trajectories involving tools such as LSP, MCP, or AST-grep were probably scarce in the dataset.

This lack of representation may also indirectly limit the effectiveness of subsequently generated synthetic data.