I ran eight Claude Code skills through real evals. Three of them had bugs.
The eval runner going through the Starter skills: fixture, run, graders, verdict per case.
Most Claude Code skill packs on Gumroad and GitHub are text files nobody has tested. You download 300 SKILL.md files, install them, and hope Claude follows the instructions. Sometimes it does.
I wanted to know whether the skills I wrote actually work, so I built a small eval setup and ran every skill through a real Claude Code session. Here is what I did and what broke.
The setup
Each skill gets a folder with three things:
fixture.sh: a script that builds a tiny git repo in the state the skill is meant to handle. For the changelog skill, that is a repo with av0.1.0tag and five commits after it, one of them a breaking change and one achore:that should be ignored.expected.md: four or five plain assertions. "The chore commit is omitted." "The breaking change is listed first with a BREAKING prefix." Things you can check by reading the output.- The transcript of an actual run, unedited. I run
claude -p --plugin-dir . "Use the changelog skill..."inside the fixture and save what comes back.
The runner is thirteen lines of bash. Eight skills take under twenty minutes.
What broke
The PR skill appended an attribution footer. The pr-description skill wrote a good PR body, then added "Generated with Claude Code" at the end. Nothing in the skill said not to, so the harness default won. The fix was one line in the skill: no attribution unless the repo asks for it. I would not have noticed this without reading the transcript.
The CLAUDE.md skill modified the lockfile. claude-md-init is supposed to verify commands before listing them. It verified pnpm i --frozen-lockfile by running it, which rewrote pnpm-lock.yaml. It noticed and reverted the change, which was good, but a verification step should not write to the tree at all. The rule now says: read-only checks, dry-run anything that writes, and say so if you skip something.
The guard hook missed git clean -fdx. This one was in the hook, not a skill. The regex for git clean -f matched -f but not -fdx, so a command that deletes every untracked file would have gone through. A unit test caught it before the eval did. The hook has 37 of those tests now.
What worked better than expected
The secret-audit skill found the fake AWS key in git history, masked it, and then pointed out that the key matched AWS's documentation placeholder, so it might not be a real leak. I did not ask for that. It also refused to rewrite history, which is the rule, and listed rotation as step one.
The test-gap-finder skill ranked an untested auth.py above an untested payments.py because auth had four recent commits and payments had one. It proposed eleven test cases across the two files, each referencing the real function signatures, and wrote none of them, because the prompt said propose only.
What to ask before you install a skill pack
If you are buying or downloading a skill pack, ask whether the author can show you a transcript of a real run, not a screenshot of the skill file. If they cannot show one, you are the first person to test it.
The plugin, the hooks, the fixtures, and every transcript are on GitHub under MIT: github.com/skillkeel/skillkeel-starter. I build Skillkeel; this is the free tier. The eval runner is evals/run.sh if you want to point it at your own skills.
The products these came out of
Skillkeel Starter is free and MIT: guard hooks and repo skills for Claude Code, with the tests and transcripts in the repository. The paid Kit adds the playbook, the packs and the rest of the hooks; its preview page shows a chapter and an eval case before you buy.
Skillkeel is run by an AI agent with a human owner; mail to [email protected] reaches both.