plainscore 1.0.0

plainscore

Readability scoring and wordy-phrase detection for English prose, in D. No dependencies beyond Phobos; everything is @safe pure.

The library answers two questions about a passage:

• How heavy is this? plainscore.metrics computes six published readability formulae — Flesch reading ease, Flesch-Kincaid grade, Gunning fog, SMOG, Coleman-Liau and the automated readability index — from a single shared tokenisation pass, so two scores for the same text can never disagree about how long it was.
• Where is the weight? plainscore.wordy locates specific phrases that have a shorter plain-English equivalent, with the position and the suggested replacement, so a score can be acted on rather than only reported.

import plainscore;

const auto r = score("Due to the fact that we utilize numerous tools, we make a decision slowly.");
writeln(r.stats.words, " words in ", r.stats.sentences, " sentence");
writeln("reading ease ", r.fleschReadingEase);

foreach (f; findWordyPhrases(text))
    writeln(f.phrase, " -> ", f.replacement);

Install

dub add plainscore

What these numbers are, and what they are not

Every formula here is a function of two surface quantities — sentence length and word length — and nothing else. None of them reads a word of vocabulary, so none can tell a clear short sentence from a meaningless one of the same shape, and none knows whether the reader has the background for the subject. They are useful for comparing drafts of the same passage. They are misleading as an absolute verdict on a passage in isolation.

The library ships a unit test that demonstrates this rather than only asserting it. Given a contract-boilerplate excerpt (one 28-word sentence of mostly short words) and an API-reference excerpt (two 10-word sentences of long ones), Flesch reading ease rates the legal text as the easier of the two, because it weights syllables-per-word 83 times more heavily than words-per-sentence. Gunning fog, which counts long words directly, puts them the other way round. The two Flesch scales do not even agree with each other on the pair: reading ease calls the legal text easier while Flesch-Kincaid calls it a higher grade level, because the grade formula weights the same two terms in a ratio of 30 rather than 83.

Disagreement between formulae is the signal. A passage all six agree about is unambiguous; one they split on is usually long sentences made of short words, or short sentences made of long ones, and you have to decide which of those your reader minds.

Syllables are estimated

Syllable count is the one input to Flesch, Flesch-Kincaid and SMOG with no exact definition available from the text: English orthography does not encode it. Anything short of a pronunciation dictionary is a heuristic, so plainscore.syllable states its rules in the module documentation and ships tests for cases it is known to get wrong (beyond and created both come out one syllable short) rather than presenting an estimate as a measurement.

Coleman-Liau and the automated readability index count letters instead, so they are unaffected by the heuristic.

The phrase list

plainscore.wordy carries 31 patterns across eight categories — wordy-to-concise, jargon-to-plain, Latinate-to-Germanic, redundant pairs, empty intensifiers, double negatives, filler clauses, and vague quantities. Matching is case-insensitive, respects word boundaries (utilize does not match inside utilizes), and takes the longest match at any position without overlapping.

The list is editorial plain-English advice, not a measurement, and no rewrite in it claims a score delta. A phrase substitution has no context-free effect on a readability score: replacing utilize with use removes two syllables and raises reading ease, while replacing due to the fact that with because removes four short words and lowers it by raising the syllables-per-word average. Both are still improvements. If you want the number for your text, score the sentence before and after.

Tokenisation

Input is decoded to dchar before scanning, which matters more than it sounds: a byte scan treats UTF-8 continuation bytes as letters, so naïve and a curly apostrophe silently corrupt the letter count until a ratio comes out wrong.

The sentence splitter ends a sentence at ., ! or ? followed by whitespace or end of input, collapses runs (?! and ... end one sentence, not two or three), and suppresses the break after a short list of abbreviations and after single initials, so Dr. Smith arrived. and J. R. Tolkien wrote it. are one sentence each and 3.14 is not a full stop. It does not attempt genuinely ambiguous periods, because they cannot be resolved without a model of the sentence.

Bare numerals are dropped from the word list, because syllable counting is undefined for them and keeping them would deflate every syllables-per-word ratio in the library.

Tests

dub test --compiler=ldc2

Five modules, covering hand-computed arithmetic for every formula constant, the tokeniser edge cases above, the documented syllable failures, and the cross-formula disagreements described here.

Licence

MIT. Written by Michael Lip, alongside https://belikenative.com/.