Tool evaluation goes wrong when it produces an impression. You try something, it seems clever, you buy it, and three months later you have stopped opening it. A method that produces numbers rather than impressions isn't much more work, and because it's comparable between tools it lets you actually choose.
Setting up
- Copy one middle chapter into a separate file. Never test on your master.
- Plant errors you know about, and write down what and where.
- Choose errors that match your worries, not generic ones.
- Run the tool.
- Record the four numbers.
- Repeat with the next tool, on the same file.
Step one matters practically. Testing on a copy means a tool that alters your file, or a paste that goes wrong, costs nothing.
The four numbers
| Number | What it tells you |
|---|---|
| Planted errors found | Whether it does what it claims |
| Useful findings you did not plant | Its actual value, which can be higher than expected |
| Findings dismissed | The noise you will live with |
| Time taken | The cost per use, which compounds |
Watch the second number. A tool that misses your planted errors but surfaces three genuine problems you hadn't noticed is more useful than the numbers suggest, and this method captures that.
Comparing tools
Run each on the same chapter with the same planted errors. The comparison is then like for like, which no amount of reading reviews provides, and it takes an hour per tool.
The Novel Kit in this method is straightforward: plant a name variant and a term variant, run the scan, and check both appear with their counts. It won't find a doubled ordinary word or a homophone error, which is also information, and it's stated rather than discovered.
A practical next step
Prepare your test chapter once, with its planted errors, and keep it. Every future tool evaluation is then an hour rather than an afternoon.
Sources checked
Frequently asked questions
Why my own book?
Because a demonstration text is chosen to show the tool at its best, and your prose is what you will actually run through it.
Which chapter?
A middle one, of typical density, with your usual proportion of dialogue and invented vocabulary.
What am I measuring?
Four numbers: planted errors found, useful findings, dismissed findings, and time taken.