DictionaryBoundaryStrategy#

class scikitplot.corpus.DictionaryBoundaryStrategy[source]#

Dictionary-based word boundary detection for no-space scripts.

Covers:

  • Thai — PyThaiNLP newmm engine (pip install pythainlp).

  • Lao — PyThaiNLP lao engine.

  • Khmer — khmer-nltk (pip install khmer-nltk).

  • Han (Chinese) — jieba (pip install jieba).

A custom callable registered under "<script_value>_boundary" (e.g. "thai_boundary") in the tokenizer registry takes priority over the probe chain.

Parameters:
(none)

Notes

User note (Q3): Per architecture invariant Q3, all known languages are attempted. If no dictionary library is available for the detected script, a logger.warning is emitted and GraphemeClusterStrategy is used.

Developer note: Thai has no sentence-terminal character; sentence splitting for Thai is handled here via PyThaiNLP sent_tokenize.

segment(span, config)[source]#

Segment no-space span using dictionary lookup.

Parameters:
spanScriptSpan

Thai, Lao, Khmer, or Han script span.

configWritingSystemAdapterConfig

Adapter configuration.

Returns:
list[Chunk]

Dictionary-word or grapheme-cluster chunks.

Parameters:
Return type:

list[Chunk]