Evaluating a tool by reading its feature list tells you what its makers chose to emphasise. Evaluating it by running it on a chapter tells you what it does with your prose. The difference is the whole of this article, and the method takes an hour.
The test
- Take one chapter of your own manuscript, ideally a middle one.
- Plant three errors you know about: a name spelled differently, a repeated word, and whatever else you want checked.
- Run the tool.
- Count which planted errors it found.
- Count the findings you would act on among the rest.
- Count the findings you dismissed.
- Note how long steps three to six took.
Those four numbers, found errors, useful findings, dismissals and time, are what you are buying. They are comparable between tools and they are about your writing rather than about a demonstration text.
What to plant
| Planted error | Tests |
|---|---|
| A name spelled two ways | Consistency checking on invented vocabulary |
| A doubled word | Basic proofreading |
| A homophone error | Whether it goes beyond spelling |
| An inconsistent hyphenation | Style consistency |
| A changed eye colour | Whether continuity claims are real |
| A term used once | Whether the report surfaces the tail |
Row five is the one that separates claims from capability. A tool advertising continuity checking should find a description contradiction planted a few pages apart, and most don't. Better to find that out before paying.
The other half of the evaluation
Alongside what it finds, ask what it keeps: whether your text is stored, for how long, and whether it is used for training. Those questions are in questions to ask a writing tool about privacy, and they matter as much as the findings.
The Novel Kit can be evaluated exactly this way: plant a name variant, run the scan, and see whether both spellings appear with their counts. It's a yes or no test, which is the advantage of a tool with a narrow stated scope.
A practical next step
Plant three errors in a chapter and run whatever tool you are considering. An hour, and you'll have an answer rather than an impression.
Sources checked
- How to test a tool on your own book
- Free trials and what to test
- Questions to ask a writing tool about privacy
Frequently asked questions
What is the best test?
Plant an error you know about, run the tool, and see whether it finds it. That converts evaluation into a yes or no.
What should I not judge on?
The demo, the marketing copy and the feature list. All three are designed to look good and none of them is your book.
How long should evaluation take?
An hour with one chapter. Anything longer and you are exploring rather than evaluating.