Reference
Pasted text
import { … } from "@johnmorrisdotca/hikidashi/extract";2.2 kB minified and gzipped, with everything it imports (measured with esbuild)
抽出: Japanese words and kanji pulled out of pasted text.
Somebody pastes a page of a book, a lesson handout, a chat message, and wants the kanji and the words out of it. Two things follow from "somebody pastes anything".
The first is safety. The text is never stored, never rendered as anything but text, and never trusted for its length: it is cut to a cap before any work is done, control characters are dropped, and what comes back is not the text but words your own dictionary recognised and single kanji, a fixed shape of strings. Nothing a paste can contain survives that as anything but a kanji or a word you already knew.
The second is the reading itself: the runs of Japanese are found, the words in them are matched
longest-first against the dictionary you pass in as known, and every kanji is taken whether it
belongs to a matched word or not. A verb or adjective is matched as it is written (行きます, 高くて)
and returned as the dictionary form known lists (deinflect), so a sentence gives 行く rather than
the noun 行き that its stem happens to spell. The package carries no dictionary of its own: known
is a Map from each word to the kinds of word it is, from yours.
Exports
| Name | Kind | What it is |
|---|---|---|
EXTRACT_LIMITS | constant | What a paste is held to: read to 20,000 characters and then cut, words of up to 8 characters, and at most 4,000 candidates asked of a dictionary. |
extractFromText | function | The words and kanji a paste comes to: the words known lists, longest match first, and every kanji in the text. |
ExtractInput | type | What to read, and the dictionary to read it against: the argument of extractFromText. |
ExtractResult | type | The words found, then the kanji, each once, in the order they first appear; and what the paste held. |
KnownWords | type | The words your dictionary confirmed, each with the kinds of word it is. |
sanitizePastedText | function | The text as it will be read: control and invisible characters (and line and paragraph separators) turned into spaces, then cut to EXTRACT_LIMITS.characters. |
wordCandidates | function | Every substring the dictionary might know, so one question can be asked about the whole paste: look all of them up at once, and pass the ones found as known. |