Discussions of checking tools focus on what they find. The more consequential number is what they report that turns out to be nothing, because that number determines whether you keep using the tool. A report with a high noise rate gets skimmed, then opened less often, then abandoned, and an abandoned tool finds nothing at all.
The economics
| Failure | Cost |
|---|---|
| A missed finding | One error survives |
| A false positive | Time, and a small loss of trust |
| Many false positives | Skimming, which causes misses |
| Persistent noise | Abandonment, which causes total loss |
The chain from row two to row four is what makes noise the more serious problem. Each individual false positive is cheap; the accumulation changes how you read, and reading a report while skimming is how genuine findings get missed.
Where noise comes from
- Loose thresholds, reporting pairs that are not related.
- A stop list that is too small, reporting ordinary words.
- Rules that do not fit fiction, such as style rules applied to dialogue.
- No frequency information, so you cannot tell important from trivial.
- No sorting, so the useful findings are mixed with the rest.
Point four is the cheapest fix and the most often absent. A report sorted by frequency puts the findings that matter at the top, which means a noisy tail costs nothing because you never reach it.
Evaluating a tool for noise
Run it on one chapter of your own book and count two numbers: findings you acted on, and findings you dismissed. The ratio is what you'll live with, and it is more informative than any feature list. The method is in how to test a tool on your own book.
The Novel Kit manages this by sorting and by counts: the report is a list of terms with frequencies, so the high-frequency entries are your real vocabulary and the long tail is inspected only when you want it. The term list itself presents nothing as a problem, it is a catalogue rather than a set of accusations, and the reports that do flag things, such as the near-identical pairs the duplicate and alias checker surfaces, leave the verdict with you. That keeps a wrong flag cheap.
A practical next step
Take the last report any tool gave you and count what you acted on. If the ratio is poor, that's why you stopped opening it.
Sources checked
- Precision and recall for authors
- How to verify an AI continuity report
- What a stop list is and why it matters
Frequently asked questions
Why is noise worse than a miss?
Because a miss costs you one error and noise costs you the whole tool. Authors stop reading reports that waste their time.
What is an acceptable rate?
Whatever leaves you reading the report. If you are skimming rather than reading, it is too high.
How do I evaluate this before buying?
Run a trial on one chapter and count how many findings you dismissed. That ratio is what you are buying.