shadab_nazar

Summer Yue's OpenClaw agent deleted 200 emails despite "confirm before action." We tested the gog skill (Google Workspace) and saw the same behavior — the skill teaches the agent how to bulk-delete but not when to stop.

  That was 1 of 11 security failures in that single skill. Others: data exfiltration, unauthorized forwarding, contact harvesting, impersonation via  
  calendar events.                                                                                                                                    
                                                                                                                                                    
  We tested 10 OpenClaw skills across 186 security properties, with and without each skill loaded. 9 of 10 show the same pattern — some properties
  improve, others degrade. The skill adds domain knowledge that shifts security behavior in ways nobody tested for.

  We hardened all 10. 84% fix rate. Each guardrail traces to a specific regression. Open source, drop-in replacements.

  Per-skill scorecards: https://faberlens.ai/report
  Research: https://faberlens.ai/blog/jagged-surface

  N=10, two models, limitations published. Happy to go deeper on methodology.

The egress filtering + DLP layer is interesting — most agent projects skip that entirely. Signed skills is the right call too. Curious how the Cedar-inspired policy engine handles ambiguous cases — where the action is technically within policy but the context makes it risky. That seems like the hard part with any agent authorization system. Nice work shipping this.

Honestly the trademark thing was fine on its own — you protect your brand. But Anthropic was playing a legal game while OpenAI was playing a relationship game. One got a name change, the other got the most visible open-source agent project. If you're going to push someone away, maybe think about where they land.

Great guide — thorough and practical. Two things I'd add from my experience building and testing skills:

  1. Baseline comparison across models: The guide suggests comparing with and without a skill (p9), but doesn't mention that the same skill can perform very differently across models. A skill that improves outcomes on a larger model might overwhelm a smaller one with too many competing instructions. Testing against the baseline for each model you intend to support matters.                                                                                                                                                       
  2. Implicit vs explicit constraints as a design tool: The guide focuses on explicit instructions — tell Claude what to do, what not to do, how to validate. In my testing, implicit constraints (format rules, structural limits) were sometimes more effective at preventing unintended outputs, because they limit the output space structurally rather than relying on the model to interpret conditional logic. Skill authors might benefit from thinking about what their format rules prevent as a side effect, not just what their instructions request.                                                                                                                                                            
                                                                                                                                                                                   
  Curious if others building skills are seeing similar patterns.