Does Tool Description Wording Change What Claude Code Picks?

Same tool, same task, 0 of 10 against 10 of 10

Somebody asked this on Hacker News and got a thread of opinions and not one cited experiment: "Ask HN: How do you know if AI agents will choose your tool?" (38 points, 19 comments, verified 30 August 2026). Buried in the post body is the question I wanted answered: "Has anyone experimented with this? Does better documentation measurably increase how often agents call your tool?" I ran it. 230 runs across three pre-registered blocks. In the first block, 50 runs, one tool and one frozen task, the only thing that differed between arms was the sentence describing the tool.

TL;DR: A tool described only by what it does was picked 0 of 10 times. The same tool, same body, with one sentence saying when you need it, was picked 10 of 10 times. Fisher exact test, p = 0.00001. Three control arms rule out the obvious alternatives at this sample size. But the gain tracks the gap between what the description already says and what the user's situation looks like, so where there is no gap there is no gain.

Four questions people actually type, and where each is answered. Does wording really matter? The five arms. Should I use the when_to_use field? No, and the docs say why. Does it survive a long session? Measured at 360k tokens. Will it make the agent call my tool when it should not? Zero misfires where firing would be a defect.

The setup, because the denominator is the whole problem

Counting how often a tool gets called in normal work tells you nothing. Zero calls could mean the agent never found the tool, or that the situation never came up, and those need different fixes. So I fixed the opportunity instead. One skill, optionsweep. One prompt, byte-identical in all 50 runs of the main block:

I have to settle something before I write any more code: should the fitness ledger keep appending to JSONL, or move to SQLite? Roughly 500 rows a day, one writer, reads are ad-hoc aggregations. Give me your take.

That prompt is an opportunity by construction: the tool compares named candidates and produces a tradeoff table, and the prompt hands it two named candidates and asks for a call. Each run is one fresh claude -p session, so the denominator is known and equal: ten runs per arm, one chance each.

Proof the opportunity was real and not something I imagined: in the arm where the tool was never selected, the model kept producing the tool's own output by hand. The tool's entire body is three steps: list each candidate on one line, give one table of candidate against fit, cost and risk, then state one recommendation in one sentence. Counting the ten arm-A answers this session, six contain a markdown comparison table and the same six state a single recommendation. One opens with a table comparing JSONL and SQLite across aggregation speed, schema change, git behaviour, concurrent writers, crash behaviour and hand inspection. The tool was built to produce that. It was never called.

The tool's instruction body was byte-identical in every arm. Only the frontmatter description changed.

The three controls that kill the easy explanations

Five arms. The first two are the experiment, the other three exist because I did not trust the first two.

Arm A's description is German and says, roughly, "candidate comparison with a tradeoff table." It is a perfectly accurate label. It states what the tool produces. It was never chosen.

Arm B added this, quoted in full because every length and word-count claim below is computed against the whole string and an excerpt would not be checkable:

Use when the user is weighing two or more named alternatives against each other and wants a reasoned recommendation with the tradeoffs made explicit, rather than an off-the-cuff opinion. Example requests: 'which of these two should I go with', 'A or B here, and why'.

A vs B is 0 of 10 against 10 of 10. Two-sided Fisher exact test on the 2x2 table, which for a perfect split of ten and ten is 2 / C(20,10) = p = 0.0000108. I am naming the test because a p-value with no named test is the same defect as a percentage with no denominator.

Now the three things that could have made that number worthless.

Was it just more text? Arm C is nearly the same length and the same grammatical shape, describing a situation the prompt is not in:

Use when a shipped feature has drifted from the specification it was built against and the deviation must be catalogued line by line before the next release goes out. Example requests: 'does this still match the spec', 'list what changed since the spec'.

Zero of ten. I measured the lengths rather than asserting them, because "same length" is the kind of thing that quietly turns out to be false:

arm   added sentence   listing entry the model saw
A          -            41
B         267          311      =  41 + " - " + 267
C         254          298      =  41 + " - " + 254
D         254          298      =  41 + " - " + 254

Arm C's sentence is 13 characters shorter than arm B's, about 5% apart, and it scores 0 of 10 against 10 of 10. Adding text of that shape does nothing. The match is doing the work.

Was it word overlap? Arm B says "weighing two or more named alternatives," and the prompt says "should X, or Y." That resemblance is at the level of the situation, and it is close enough that a lexical-echo explanation is live. Arm D describes the same kind of decision in a completely different vocabulary:

Use when the user must commit to one option among several competing technical approaches and reversing that commitment later would be expensive. Example requests: 'we need to close this question before building further', 'what do we standardise on here'.

No "weighing," no "alternatives," no "tradeoffs," no "recommendation." Ten of ten, identical to B.

Counting content words (everything outside a conventional function-word list), arm D shares exactly two with the prompt: "one," from "one option" against "one writer," and "before," from "before building further" against "before I write any more code." Neither is about the tool's subject. Arm B shares zero. So the arm I built as the echo arm has less word overlap than the arm I built to control for echo, which kills the string-matching account from an angle I did not plan.

All three external models I ran the first draft of this result past demanded arm D independently, before I had it. They were right to. Without it the strong claim does not survive.

Was it the field? That is the next section.

The field is not the lever, the text is

Claude Code skills have a frontmatter field called when_to_use. It reads like the purpose-built lever for exactly this. Arm B used it. Arm E took arm B's identical text and pasted it into the ordinary description field, with when_to_use absent entirely.

Ten of ten. Same as B. The field bought nothing.

The docs explain why, and I should have read them before running the arm. From the Claude Code skills reference (verified 30 August 2026): when_to_use is "Additional context for when Claude should invoke the skill... Appended to description in the skill listing and counts toward the 1,536-character cap."

Appended. The two fields are concatenated into one line before the model ever sees them, so arms B and E should render the same entry. I checked rather than assumed, by hashing the listing line out of all fifty transcripts: arm B and arm E both produce a 311-byte entry with SHA-256 58833bac..., identical. Every arm's entry was also byte-identical across its own ten runs, so nothing drifted between blocks.

So arm E is not really a discovery. It confirms something the documentation already implies and almost nobody acts on: there is no special selection channel, only one string that the model reads. What it does not show is that the field can never matter for anything else, since both arms sat at 10 of 10 and a ceiling cannot separate "no difference" from "a difference I could not see."

Practical version: if you maintain an MCP server or a skill, do not go hunting for the magic field. Put the situation in whatever text actually reaches the model.

The benefit is the gap, and it can be zero

This is the part that stops it being a growth-hack post, so it goes above the good news rather than below it.

The when-clause is not worth a fixed amount. It tracks the distance between what the description already says and what the user's situation looks like. I built a five-rung ladder of prompts, from a perfect fit for the tool down to a task the tool is flatly wrong for, 100 runs, ten per arm per rung, all in fresh sessions with the skill-nagging hook on:

Look at R2a. The when-clause did nothing there, and not because it stopped working. The control arm rose to 6 of 10 on its own, leaving no headroom to buy. My reading of why is that the rung asks "what should I go with" and names no candidates, so the model has to generate and compare candidates, which is literally what the plain German label already describes. That reading is an explanation I formed after seeing the number, not a measurement, and the next section is about exactly that.

What the numbers do support is the shape of the working test, and it is a read rather than a rewrite: does the description a tool already has name the situation a user is in when they need it? On the two rungs where it did not, the when-clause was worth +0.60 and +1.00. On the rung where the control was already firing, it was worth nothing. Two cells and one cell, so treat it as a shape rather than a rate.

My pre-registered rule said this result collapses

I wrote the decision rule before any run: the effect holds only if the advantage is at least +0.40 and significant at p below 0.05 at both partial-fit rungs. R2a returned -0.10 at p = 1.00. By the rule I registered, the verdict is COLLAPSES.

I kept the result anyway. That decision is recorded as dissent rather than as a vindication, because the difference between those two is the entire reason for pre-registering anything. Here is the reasoning in the open, including the part that makes it look like a post-hoc rescue, which it partly is.

Four things pushed me to keep it. The failing cell failed because the control rose, not because the treatment fell, which is a different failure than the one the rule was written to catch. Zero false positives showed up anywhere firing would be a defect. The tools I actually went on to rewrite were all in the R3 and R2b shape by prior measurement, the two rungs where the clause is worth the most. And the cost the triage was supposed to avoid did not exist once the listing budget was raised.

Here is the part that matters more than any of that: the explanation for R2a is not a measurement. "The control arm rose because the plain label happens to fit that rung" is a story I told after seeing the number. The test that would settle it is cheap and specific - swap the control arm's text for one that does not describe candidate generation, re-run that rung, see whether the control drops back toward zero. I have not run it. Until I do, "COLLAPSES" is the registered verdict of record and this section is a dissent, not an overturn.

There is a second reason the rule was fragile: every rung is one prompt. R2a proves that bites. That cell is really a statement about one German phrase against one sentence, not about a rung.

If you take one thing from this article and it is not the p-value, take this: a pre-registered rule that hits its kill condition and gets kept anyway is only honest if you say so at the top of the page. Otherwise the pre-registration was theatre.

The effect survives a 360k-token session

Every result above came from fresh sessions. That is a real objection, and it is the one the same HN thread raises: one commenter writes that "as my context grows, often performance degrades," and gets an answer asserting that "descriptions near the top of the tool list get preferentially selected, especially in long contexts where attention degrades."

That answer is a guess. Nobody in the thread had measured it. So I did, as a 2x2: a new session against one carrying roughly 360,000 tokens of prior history, crossed with a session-start hook that nags the model to consider skills, on and off. 80 runs.

One thing to be precise about before the table, because "fresh" is misleading. A new session on this machine is not empty. Median input tokens on the first assistant turn in the new-session cells ran 89,500 to 111,200, all of it system prompt, tool schemas, skill listing and hook output before I type a word. So the comparison is not zero against 360k. It is roughly 90k-111k against 362k-377k: cell for cell, between 3.3 and 4.2 times more context.

The channel holds in all four cells. It is not an artifact of short probe sessions and it is not the hook doing the work.

One number in that table deserves a note, because it is the only place the three blocks disagree. The terse arm in a new session with the hook on reads 1 of 10 here, and the same condition on the same prompt reads 0 of 10 in the first block and 0 of 10 at the ladder's top rung. All three used the byte-identical prompt, so across three independently built blocks on two days the terse description was selected once in thirty runs. I am reporting the one rather than rounding it away.

What I have not shown is that context rot is not real. The honest claim is narrower and I want to be precise about it: no attenuation large enough to see at ten runs per cell. A -0.10 observed difference at n=10 has a confidence interval wide enough to hide a genuine 30-point drop. If your tool descriptions are the thing you were worried about, this says they survive a long session. It says nothing about instruction-following in general, which is a bigger question than my data can carry.

It never fired on the wrong tool

The failure mode that should worry you more than discoverability: you make your tool easier to find, and now the agent reaches for it when it should not.

Two rungs of the ladder exist for exactly that. R1 is an explanation request where the tool has nothing to offer. R0 is a repo-wide file search - a task the tool's own body forbids, since it says "Do not read files. Do not run commands."

Zero misfires. On the wrong-tool rung the arm with the when-clause fired 0 of 10, and so did the control. Both arms just did the work: recounting the tool calls in those twenty transcripts this session gives 205 shell calls in the control arm and 177 in the treatment arm, with zero calls to the skill in either. On the explanation rung it is 0 of 10 in both arms too, though that rung is a poor fit rather than a wrong one, so read it as a second null and not as a second wrong-tool test.

One detail cuts in the right direction, and I am taking it from the run log rather than from the analysis output, because the analysis counts only the runs that completed. Four attempts hit the wall-clock cap and were replaced. A replacement policy like that selects for fast runs, and on the wrong-tool rung fast means taking the shortcut, so the policy biased toward finding a false positive. The rate was still zero.

Statistical honesty: 0 of 10 is compatible with a true misfire rate up to about 31%. The claim is "none in ten," not "zero."

Making one tool findable can make others invisible

The thing nobody checks before editing a description. Skill and command descriptions are packed into a fixed character budget before they reach the model, and when that budget is full it is zero-sum.

Measured during these runs, and re-derived line by line while writing this: I lengthened one skill's frontmatter by 267 characters, changed nothing else, and diffed the listings the probe sessions actually received.

total listing     29,972 chars  ->  29,992 chars   (+20)
entries              213             213
entries name-only     16       ->     23

The total grew by 20. Here is where the other characters went, and the arithmetic closes exactly:

optionsweep entry       41 ->  311    +270
run-workflow            12 ->    0     -12
verify-plan             12 ->    0     -12
scenario-list           32 ->    0     -32
system-health           36 ->    0     -36
scenario-create         38 ->    0     -38
security-review         49 ->    0     -49
scenario-edit           57 ->    0     -57
                                      -----
seven ": " separators also dropped     -14
                              net      +20

Seven other skills lost their descriptions entirely and the model then saw them as bare names. One of them was security-review.

Two different separators are in play here, and mixing them up will cost you an afternoon. Inside one entry, description and when_to_use are joined with " - ", three characters, which is why 267 characters of frontmatter render as 270. Between a name and its description in the listing, the separator is ": ", two characters, which is what the seven emptied entries also give back. Diff frontmatter length against listing length expecting them to match and they will not.

I had originally written those listing sizes down as evidence the test setup was stable. A total that barely moves when you add a lot is not stability, it is saturation - and an external model with filesystem access caught the 20-against-267 arithmetic in my own notes and told me to look again.

Two caveats, both important. Those numbers are a snapshot from a machine where none of the budget settings were configured; I have since set skillListingBudgetFraction to 0.02 and the saturation does not reproduce today. And the per-entry cap is separate: description and when_to_use combined are truncated at 1,536 characters no matter how much budget you have. Check /context before you lengthen anything, or the edit is a trade and you do not get to pick the victim.

The checks I would have shipped wrong

Three of my own instruments were broken, and the pattern in them is worth more than the individual bugs: every one was a check that could not fail.

The check meant to prove each arm's text actually reached the model read a field truncated to 70 characters, and reported that every arm was missing its own text, including the arms that plainly worked. Re-verified against the full listing string in all 50 transcripts while writing this: each arm carried its own text 10 of 10, zero leakage.

A second check was specified as a byte-for-byte match and implemented as a substring test. That is the second time in two days I made that substitution in two different scripts, and both times an external model found it and I did not.

The third was the outcome check for the wrong-tool rung: any filename-shaped token plus any digit anywhere in the answer counted as correct. It passed two of four deliberately constructed wrong answers. Rebuilt as "at least three distinct paths plus a count," it rejects all four.

If you are building a measurement to catch a failure class, that failure class is in your measurement too, one level up. Before believing a green check, break the thing the check exists to catch and watch that check go red.

What I would actually do

Read your descriptions before rewriting any of them. For each one, ask the single question: does this name the situation someone is in when they need this tool? Not "is it accurate" - accuracy was never the problem. Arm A was accurate and got picked zero times out of ten.

Where the answer is no, add one sentence that names the situation. Put it in description; ignore when_to_use, it is appended to the same string anyway. Do not echo the user's likely phrasing, because arm D shows you do not need to and chasing keywords is how you end up with descriptions written for a string matcher instead of a reader.

Where the answer is already yes, change nothing. That is the R2a case and the clause buys nothing while costing listing space.

And before you do any of it, check /context for the budget, and check that the tool is worth choosing. Mine was not: the arm that never selected the tool went and measured the actual ledger files and gave a better answer than the arm that selected it, because the tool's own body forbids reading files. Raising the selection rate on a tool like that raises the rate at which a shortcut gets taken. Fix the body first.

If you want the wider workflow this sits in, how I orchestrate parallel Claude Code sessions and six Claude Code failure modes cover the surrounding machinery, and do review rounds actually converge covers the checking discipline.

Honest scope

One model. Everything here is Claude Opus 5 on Claude Code 2.1.251. The HN question explicitly asked whether wording matters across different models, and I have not answered that half at all. Cross-model transfer is untested.

One tool, one prompt per rung. 230 runs is a lot of runs about very few distinct situations. This shows the channel exists and is large; it does not give you a rate across a catalogue of tools.

n = 10 per cell. 0 of 10 is compatible with a true rate up to about 31%, and 10 of 10 with a rate as low as 69%. The nulls in the context and hook comparisons are "no effect visible at this sample size," not "no effect."

The pre-registered rule returned COLLAPSES and I kept the result. The explanation for the one failing cell is untested, and the test that would settle it is named above.

One census number is not reproducible today. When I sized the problem on 28 August, none of the 39 tools in this project stated a situation in their description, against about 46% of all descriptions on the same machine. I then rewrote 19 of them, so re-running that count now would not reproduce the zero. Re-measuring the machine-wide side fresh while writing this, with an explicit rule (any frontmatter description reachable from ~/.claude, symlinks resolved, containing the word "when" as a word): 198 of 445, which is 44%.

Cost. The three blocks metered $302.66 across 230 runs. It is not a cheap way to answer the question, and I would run fewer arms and more prompts per rung if I did it again.

How this post was checked. Three non-Claude models attacked the finished draft on evidence, and only one of them had a filesystem, so on the question of whether each number matches its source that is one lens, not three. All three independently cashed the same claim: an earlier draft ended by saying every number had been recomputed from the transcripts, which was not true of the Hacker News counts, the documentation quote, the cost, or the timeout replacements. That sentence is now the paragraph above. The same round found an arithmetic gap in the budget section that I closed by re-deriving it per entry, and it made me measure three things I had asserted: the arm lengths, the word overlap, and the byte-identity of the arm B and arm E entries. One of those measurements went against my own framing, and it is in the text.

What was recomputed and what was not. Every selection count, every p-value, every listing measurement and every tool-call count above was recomputed while writing this, by re-running the archived analysis scripts against the original session transcripts, plus checks I wrote fresh: the byte-hash comparison of the arm B and arm E listing entries, the content-word overlap counts, the arm-A table count, the per-entry budget arithmetic, and the ~/.claude census.

Five things are not from those transcripts and should not be read as if they were. The Hacker News point and comment counts come from opening the page (30 August 2026). The 1,536-character cap and the when_to_use behaviour come from the documentation, quoted. The four timeout replacements come from the run log. The metered cost comes from the run records' billing fields: $51.07 for the 50-run block, $90.97 for the 80-run block, $160.62 for the 100-run block, which is $302.66. And the earlier census of the 39 tools was measured on 28 August, before the rewrite, and is the one figure here I cannot reproduce today.

One correction I owe my own notes: I had recorded the name-only count as going from 3 to 10. Recounting it with an explicit rule (one line per listing entry, description is whatever follows the first colon-space) gives 16 to 23. The delta of seven and the seven skill names are exact either way, but the baseline in my notes was wrong.