Why You Should Write a Skill for Your Software

 Imagine a very hypothetical situation, definitely not inspired by hundreds of Discord threads showing this exact pattern:

Sometimes, a setup can go wrong. A bouncer that installs cleanly but never blocks a thing, or a parser that reads zero lines, and you just want the shortest path out. So you do what everyone actually does in 2026: you don’t open the docs, you alt-tab to an LLM and paste the question. Answer in seconds.

And it sounds right. It hands you a packagecloud URL, a cscli command or two, a config path. You copy-paste. Sometimes it works. Sometimes the URL is two years stale, one of those confident flags doesn’t exist, and the config path moved three releases ago. Now you’re debugging the helper on top of the thing you started with.

To be fair to the model: You handed a fast improviser a question with no map, so it did the thing that setup rewards: it improvised. That’s just what you get when you ask someone to recite a manual they were only ever allowed to skim.

So the fix is simple: give it a better map: https://www.skills.sh/crowdsecurity/crowdsec-skill/crowdsec

That’s what a skill is: our own CrowdSec docs, restructured into rails the agent can’t wander off. Context7 and friends already pipe fresh reference into your agent’s context, and that helps, but reference is still something the model has to read and choose to act on. A skill goes further and turns those docs into a procedure: the exact install command, a heads-up on the cscli flags that don’t exist, and a hard rule that it won’t touch your firewall without asking. Auto-generated docs make the model better informed, while skill makes it better behaved.

Last year we shipped an MCP to author and test WAF rules for CrowdSec  (crowdsec-local-mcp), but it seems most people just want the boring part to work: install it, wire a bouncer, stop the probes. MCPs were so 2025, so this time we tried something simpler: a skill.

And building it did something I didn’t plan for: it made our own docs better and improved the release testing process. I’ll come back to that.

What a skill adds to an already powerful model

Three reasons this earns a place in your middleware chain.

Models can now hack the planet over a weekend unnoticed and solve 87-year-old mathematical problems while you watch a football match – does it struggle to set up a simple piece of software as CrowdSec? (Spoiler: yes, it did).

I ran the same experiment with a “vanilla” model (Opus 4.8 High) and then with the CrowdSec skill enabled, and here is what changed:

• With the skill, it looked before it leapt. After the initial install, it inspected what service was detected  (nginx, sshd, postfix) and if the logs were correctly ingested and parsed.
• Vanilla model did some very reasonable guessing that ended up being wrong: It reached for /etc/crowdsec/acquis.yaml to add nginx log files monitoring. But the initial setup already detected and monitored those log files, so now they’re read twice, making every scenario twice as sensitive as it should be.
• With the skill, it checked the outcome of its action before declaring victory, because the skill taught him to do so: inspect cscli metrics, run diagnostics, perform proper testing instead of “assuming”.
• Vanilla model failed to properly test if the WAF was correctly configured and curled the AppSec port without the API-key header, declaring it a success prematurely.
• Last but not least, the Vanilla model tried to whitelist itself. It ran cscli decisions add --ip <ip> --type whitelist. The command itself is syntaxically correct and even prints a “Decision successfully added”, but it does nothing: whitelist isn’t a decision type that has any meaning in a default setup.

Long story short: one setup was correct and relied on an actual check to ensure correctness; the other one made reasonable guesses that unfortunately ended up being just wrong enough. You can consider the skill as a loose form of harness – thanks to the feedback loop that cscli verification provides – and domain-specific knowledge in addition to an already powerful model.

The fine line between “reasonable guess” and “improvisation”

As LLMs will never say no, the skill is also a good way to draw limits on what we know the model would get confidently wrong if we didn’t stop it. For example, the skill explicitly states that it doesn’t know how to write WAF rules or scenarios without a significant error margin.

The skill is also a useful place to flag destructive and dangerous actions it shouldn’t attempt. I guess the internet is filled with enough “omg claude deleted database !!11” horror stories that I don’t need to expand on this.

And the best part of it is that a skill is mostly just a bunch of markdown files that dictate how it should behave in a very plain, human language.

The part that helped us, too

Here is the thing we did not expect when we started.

To build this skill, we let Claude run unchained on dedicated VMs, with the actual docs as a bible, and told it to actually do the work: bare-metal, Docker, Kubernetes, OPNsense, with the nginx, haproxy, traefik, and firewall bouncers. Install, wire, break, verify; on real boxes it was free to trash.

We also fed him a thousand or so of support threads from our Discord server as exercises to test its effectiveness.

And it kept tripping over our own documentation.

A command that had quietly changed. A config path that moved two releases ago and got left behind in the docs (that acquis.yaml the plain model reached for earlier? we still had stale references to it too). An approximation that was close enough to read past, but wrong enough to break a copy-paste. A flag that no longer existed. None of these were dramatic. All of them were the small lies that turn a ten-minute setup into a lost afternoon.

Every time the agent hit one, it was a bug report we could not argue with, because there was a broken box sitting right there proving it. So we fixed the docs. Then re-ran. Then fixed the next one.

That is the loop:

   docs  ──►  run live on real VMs  ──►  build the skill
    ▲                                          │
    └──────────  fix the docs  ◄───────────────┘

Our docs were not – are not and never will be perfect. They had the same stale corners and confident approximations that tripped the plain model up in the comparison above. The difference is that we went looking, with a tool relentless enough to find them, and now there are fewer. The skill made CrowdSec easier to install.

Beyond first-time setup

Setup is where I started because it is the wall most people hit first. The same conversation handles the two that come next.

Debug. When things go the wrong way (logs parsed but no alerts, a decision that exists but does not block), you point the agent at it and let it perform exhaustive diagnostics by relying on metrics, cscli support dump and alike.

Tune. The first false positive is a rite of passage, and the tempting fix, disabling a whole collection, trades one problem for a worse one. The skill walks you to the right suppression layer instead (allowlist, whitelist parser, or postoverflow).

Setup is just the starting point. The same principle carries through debugging and tuning: give the model the right procedure, and make it verify its work.