News for hackers. Stuff that matters.
Re:Skills Officially Comes to Codex by mikaelaast on Saturday December 20, 2025 @06:49AM (#46335496)
Are we sure that unrestricted free-form Markdown content is the best configuration format for this kind of thing? I know there is a YAML frontmatter component to this, but doesn't the free-form nature of the "body" part of these configuration files lead to an inevitably unverifiable process? I would like my agents to be inherently evaluable, and free-text instructions do not lend themselves easily to systematic evaluation.
Re:Skills Officially Comes to Codex by coldtea on Saturday December 20, 2025 @08:01AM (#46335865)
>doesn't the free-form nature of the "body" part of these configuration files lead to an inevitably unverifiable process?

The non-deterministic statistical nature of LLMs means it's inherently an "inevitably unverifiable process" to begin with, even if you pass it some type-checked, linted, skills file or prompt format.

Besides, YAML or JSON or XML or free-form text, for the LLM it's just tokens.

At best you could parse the more structured docs with external tools more easily, but that's about it, not much difference when it comes to their LLM consumption.

Re:Skills Officially Comes to Codex by Etheryte on Saturday December 20, 2025 @06:57AM (#46335535)
The modern state of the art is inherently not verifiable. Which way you give it input is really secondary to that fact. When you don't see weights or know anything else about the system, any idea of verifiability is an illusion.
Re:Skills Officially Comes to Codex by mikaelaast on Saturday December 20, 2025 @07:12AM (#46335608)
Sure. Verifiability is far-fetched. But say I want to produce a statistically significant evaluation result from this – essentially testing a piece of prose. How do I go about this, short of relying on a vague LLM-as-a-judge metric? What are the parameters?
Re:Skills Officially Comes to Codex by coldtea on Saturday December 20, 2025 @08:04AM (#46335888)
Would a structured skills file format help you evaluate the results more?
Re:Skills Officially Comes to Codex by mikaelaast on Saturday December 20, 2025 @08:45AM (#46336151)
Yes. It would make it much easier to evaluate results if the input contents were parameterized and normalized to some agreed-upon structure.

Not to mention the advantages it would present for iteration and improvement.

Re:Skills Officially Comes to Codex by coldtea on Saturday December 20, 2025 @11:34AM (#46337354)
"if the input contents were parameterized and normalized to some agreed-upon structure"

Just the format would be. There's no rigid structure that gets any preferrential treatment by the LLM, even if it did accept. In the end it's just instructions that are no different in any way from the prompt text.

And nothing stops you from making a "parameterized and normalized to some agreed-upon structure" and passing it directly to the LLM as skills content, or parsing it and dumping it as skills regular text content.

Re:Skills Officially Comes to Codex by visarga on Saturday December 20, 2025 @12:39PM (#46337925)
You 100% need to test work done by AI, if it's code it needs to pass extensive tests, if it's just a question answered, it needs to be the common conclusion of multiple independent agents. You can trust a single AI as much as a HN or reddit comment, but you can trust a committee of 4 as a real expert.

More generally I think testing AI by using its web search, code execution and ensembling is the missing ingredient to increased usage. We need to define the opposite of AI work - what validates it. This is hard, but once done you can trust the system and it becomes cheaper to change.

Re:Skills Officially Comes to Codex by JamesSwift on Saturday December 20, 2025 @04:17PM (#46339715)
How would you evaluate it if the agent were not a fuzzy logic machine?

The issue isnt the LLM, its that verification is actually the hard part. In any case, its typically called “evals” and you can probably craft a test harness to evaluate these if you think about it hard enough

Re:Skills Officially Comes to Codex by hu3 on Saturday December 20, 2025 @07:16AM (#46335633)
At least MCPs can be unit tested.

With Skills however, you just selectively append more text to prompt and pray.

Re:Skills Officially Comes to Codex by heliumtera on Saturday December 20, 2025 @12:46PM (#46337992)
Then rename your markdown skill files to skills.md.yaml.

There you go, you're welcome.

Re:Skills Officially Comes to Codex by joshka on Saturday December 20, 2025 @06:25PM (#46340677)
The DSPy + GEPA idea for this mentioned above[1] seems like it could be a reasonable approach for systematic evaluation of skills (not agents as a whole though). I'm going to give this a bit of a play over the holiday break to sort out a really good jj-vcs skill.

[1]: https://news.ycombinator.com/item?id=46338371