GraphemeClusterStrategy#

class scikitplot.corpus.GraphemeClusterStrategy[source]#

Grapheme-cluster-level fallback strategy for any script.

Splits text into individual grapheme clusters (UAX #29 \X). When unit="word" the clusters are joined in runs separated by Unicode whitespace; when unit="grapheme_cluster" every cluster is its own chunk.

This is the mandatory terminal fallback — it never depends on any optional library, so it always succeeds.

Parameters:
(none)

Notes

Developer note: Used as the fallback by every strategy that cannot load its primary optional dependency. May also be used directly for EMOJI, SYMBOLIC, UNKNOWN, and MIXED script spans.

Examples

>>> from scikitplot.corpus._chunkers._custom_tokenizer import ScriptSpan, ScriptType
>>> cfg = WritingSystemAdapterConfig(unit="grapheme_cluster")
>>> strategy = GraphemeClusterStrategy()
>>> span = ScriptSpan("abc", ScriptType.LATIN, "ltr", 0, 3)
>>> [c.text for c in strategy.segment(span, cfg)]
['a', 'b', 'c']
segment(span, config)[source]#

Split span into grapheme-cluster or whitespace-word chunks.

Parameters:
spanScriptSpan

Script span to segment.

configWritingSystemAdapterConfig

Adapter configuration.

Returns:
list[Chunk]

Non-empty list; at minimum a single chunk with span.text.

Parameters:
Return type:

list[Chunk]