skill-optimizer — Summary

skill-optimizer is a Docker-based eval workbench and agent skill for running deterministic evaluations against agent skills across multiple LLMs via OpenRouter. It ships a TypeScript CLI and a canonical skill (SKILL.md) that can be installed into Claude Code, Codex, Cursor, OpenCode, and Gemini CLI. The workbench runs each eval in an isolated Docker container with a deterministic grader — the agent cannot see the grader, hidden answers, or eval metadata during the task phase.

A case is one user-like task plus one or more deterministic graders; a suite is a matrix of cases × OpenRouter models. Results include trace.jsonl (what the agent saw, said, and did), result.json, and suite-result.json. The skill teaches the agent how to author and debug eval suites for its own skills.

Compared to seeds: no direct equivalent in the 11 seeds. Closest to spec-kit (structured validation) but inverted — spec-kit validates code against specs; skill-optimizer validates skills against eval cases. The Docker isolation makes this unique in the corpus: it's the only tool that runs agent skills in hermetic containers to produce deterministic pass/fail results across a model matrix.

skill-optimizer — Overview

Origin

Built by fastxyz (fastxyz GitHub org). Released under MIT. Supports Claude Code, OpenCode, Codex, Cursor, and Gemini CLI plugin formats. Default branch is development.

Philosophy

Skills for AI agents are like software: they need tests to verify they work reliably. The workbench provides a framework for writing "eval suites" — automated test cases that verify whether an agent skill produces correct outputs across different models. The goal is deterministic, reproducible evaluation rather than subjective judgment.

"The workbench gives an agent a skill/reference folder, an isolated /work directory, and deterministic graders."

Key insight: the agent phase is hermetically isolated — it cannot see the grader, hidden answers, or eval metadata. This prevents the eval from measuring whether the model knows the answer format rather than whether the skill works.

Design Principles

1. Prefer real CLI/API/service over mocks — mock only when you're sure the mock matches the real command surface
2. Deterministic graders — graders run after the agent with full context; agent runs blind
3. Local secrets stay local — trace.jsonl, result.json, and preserved workspace dirs are potentially sensitive if the agent prints secret values
4. MCP support — cases can define MCP servers; local server source can be hidden from the agent via separate service containers

Explicit Antipatterns

• Mock when you don't know the real CLI's behavior well enough (measures the mock, not the skill)
• One broad grader per case (prefer multiple small deterministic graders)
• Exposing grader logic, hidden answers, or /case to the agent during eval phase
• Hardcoded model refs (only openrouter/... model refs supported)

skill-optimizer — Architecture

Distribution

• Type: npm-package (TypeScript CLI) + skill-pack
• License: MIT
• Install complexity: multi-step (npm install + Docker + OPENROUTER_API_KEY)

Install Commands

# Claude Code plugin
/plugin marketplace add fastxyz/skill-optimizer
/plugin install skill-optimizer@skill-optimizer

# Cursor
npx skills add fastxyz/skill-optimizer --skill skill-optimizer -a cursor -y

# Skill-only install (any agent)
npx skills add fastxyz/skill-optimizer --skill skill-optimizer -a claude-code -a opencode -a codex -a cursor -y

# Local CLI
npm install && npm run build

Directory Layout

skills/skill-optimizer/
└── SKILL.md              # Canonical skill: authoring + debugging eval suites

src/
└── cli.ts                # TypeScript CLI (run-case, run-suite)

examples/
└── workbench/pdf/        # Example suite

tests/

docker/                   # Docker workbench image

.claude-plugin/           # Claude Code marketplace plugin
.codex-plugin/            # Codex plugin
.cursor-plugin/           # Cursor plugin
.opencode/                # OpenCode plugin
gemini-extension.json     # Gemini extension
AGENTS.md, CLAUDE.md, GEMINI.md  # Agent guidance files

Container Architecture

Docker container: /work    ← agent-visible workspace
Docker container: /case    ← case definition (agent cannot see)
Docker container: /results ← grader output (agent cannot see)
Docker mounts: references → /work (skill files visible to agent)
Optional: MCP service containers (agent sees only HTTP URL)

Required Runtime

• Node.js 20+
• Docker
• OPENROUTER_API_KEY

Target AI Tools

• Claude Code
• OpenCode
• Codex
• Cursor
• Gemini CLI

skill-optimizer — Components

Skills

CLI Commands

Case Definition Format (YAML)

name: extract-pdf-facts
task: |
  Read statement.pdf and write answer.json with the account, quarter,
  approval code, and risk flags.
graders:
  - name: answer-json
    command: node $CASE/checks/extract-pdf-facts.mjs

Suite Definition Format (YAML)

name: pdf-skill-eval
references: ./references   # Files copied into /work for agent
models:
  - openrouter/google/gemini-2.5-flash
env:
  - OPENROUTER_API_KEY
timeoutSeconds: 600
setup:
  - node $CASE/checks/create-inputs.mjs
appendSystemPrompt: |
  Keep task outputs at the top level of /work unless the user asks otherwise.
cases:
  - name: ...

Output Artifacts

skill-optimizer — Uniqueness & Positioning

Differs From Seeds

No direct equivalent in the 11 seeds. The closest conceptual analog is spec-kit's validation phase, but skill-optimizer is inverted: spec-kit validates code against specs during development; skill-optimizer validates agent skills against deterministic eval cases in isolated Docker containers. The key differentiator is Docker-based hermetic isolation — the agent runs blind, graders run with full visibility, and traces capture everything for debugging.

Also distinct from heavy3-code-audit (which runs LLM-based review) and aurite-agent-verifier (which applies static rules to code). skill-optimizer runs behavioral evals: does this skill, given this task, produce the correct output? That is a functional test, not a static analysis.

Observable Failure Modes

1. Docker dependency: Requires Docker running locally — higher friction than pure-npm tools.
2. OPENROUTER_API_KEY required: No free tier; all model runs cost money.
3. No GUI for results: Users must read suite-result.json manually.
4. Suite setup complexity: Getting references/, checks/, bin/ directories right requires understanding the isolation model.
5. Default branch is development: Install commands from README may not match latest if branch diverges.

Distinctive Opinion

Skills for AI agents are programs and should be tested like programs: with deterministic test cases, isolated environments, and reproducible results across models. The "eval workbench" pattern — separate graders from agent, run blind, grade after — is borrowed from competitive programming judges and should be the standard for AI skill quality assurance.

Target User

Primarily skill authors who want to verify their skills work reliably across models and across skill updates — not end users of those skills.