Term extraction sounds like it requires understanding and does not. It is a set of tests applied to each word in a file: is this word in the list of ordinary English, is it capitalised where a capital is not expected, how often does it appear, and where. The combination of those signals identifies the vocabulary that belongs to your book rather than to the language.
The signals
| Signal | What it indicates | Weakness |
|---|---|---|
| Not in an ordinary-word list | Invented or specialist | Rare real words are also absent |
| Capitalised mid-sentence | A proper noun | Sentence starts, and emphasis |
| Frequency | Deliberate vocabulary rather than a typo | A term used once is still a term |
| Distribution across the file | Recurring vocabulary rather than one scene | Some terms belong to one scene legitimately |
| Word shape | Unusual letter combinations | Noisy on its own |
| Adjacent words | Titles before names, for instance | Language-specific |
No single signal is sufficient on its own; a usable list comes from the combination. A word that is not in the ordinary list, appears capitalised mid-sentence, and occurs forty times, is almost certainly a name.
What this implies for the report
- Rare real words will appear. An unusual but genuine English word looks the same as an invented one.
- Typos appear too, which is a feature: a misspelling of an invented name is exactly what you want to see.
- Ordinary words will not appear, however important they are to your book.
- Classification is contextual. A term is filed as a person or a place from what the sentences around it do, being spoken by, travelled through, so a term the text never acts on stays unclassified.
- Counts are exact, since counting is arithmetic.
Point three is the boundary that matters most in practice. Eye colour, ages, distances and relationships are ordinary vocabulary and outside the scope entirely.
Why the ordinary-word list matters
The quality of the list of ordinary words determines what is reported, and it's why a stop list is a design decision rather than an implementation detail. That is covered in what a stop list is and why it matters.
The Novel Kit works this way: rules over a file, producing terms with counts and first appearances, identically on every run, with no model involved and no manuscript stored afterwards.
A practical next step
Look at a term report from your own book and find one entry that is a real English word. That's the mechanism showing its edges, and it tells you how to read the rest.
Sources checked
- Automated glossary creation explained
- Why a dictionary is not enough
- What a rule-based checker cannot do
Frequently asked questions
How does a tool know a word is invented?
It does not, exactly. It knows the word is not in a large list of ordinary English, which is a good proxy and not the same thing.
What about capitalised words?
Capitalisation mid-sentence is a strong signal of a proper noun, and it produces false positives at the start of sentences, which is why it is one signal among several.
Why does frequency matter?
Because it separates deliberate vocabulary from typos and one-off oddities, which look identical otherwise.