ScriptSegmenter#

class scikitplot.corpus.ScriptSegmenter(*, min_span_chars=1, inherit_direction=True, unknown_script_warning=True)[source]#

Segment NFC-normalised text into contiguous Unicode script spans.

Uses regex Unicode Script property matching (\p{Script=X}) rather than hard-coded codepoint ranges, so it automatically supports new scripts added by future Unicode versions.

Common and Inherited codepoints (punctuation, combining marks) attach to the preceding script span. If no preceding span exists they attach to the following span.

Parameters:

min_span_charsint, optional: Minimum grapheme count for a span to be emitted as a standalone span. Spans shorter than this are merged into the adjacent dominant span. Default 1 (no merging).
inherit_directionbool, optional: Common / Inherited codepoints inherit direction from the adjacent span. Default True.
unknown_script_warningbool, optional: Emit a logger.warning when a codepoint’s script property is not in the known ScriptType enum. Default True. The span is still produced with ScriptType.UNKNOWN.

Raises:

ImportError: If the regex (PyPI) library is not installed. Message includes the exact pip install regex command.

Parameters:

min_span_chars (int)
inherit_direction (bool)
unknown_script_warning (bool)

Notes

User note: Install regex with pip install regex before using this class. It is the only external dependency of Layer 1.

Developer note: Uses regex Unicode property syntax. All spans are produced in logical (storage) order — never reordered. ScriptSegmenter is stateless; all state lives in local variables within segment.

Idempotency guarantee:: Segmenting already-NFC text twice produces identical spans.

References

UAX #24 (Script Property): https://unicode.org/reports/tr24/ regex library: https://pypi.org/project/regex/

Examples

>>> seg = ScriptSegmenter()
>>> spans = seg.segment("Hello مرحبا world")
>>> [s.script.value for s in spans]
['latin', 'arabic', 'latin']

DIRECTION_MAP: ClassVar[dict[str, str]] = {'arabic': 'rtl', 'armenian': 'ltr', 'cjk': 'ltr', 'cyrillic': 'ltr', 'devanagari': 'ltr', 'egyptian': 'ltr', 'egyptian_hieroglyphs': 'ltr', 'emoji': 'ltr', 'ethiopic': 'ltr', 'georgian': 'ltr', 'greek': 'ltr', 'han': 'ltr', 'hangul': 'ltr', 'hebrew': 'rtl', 'hiragana': 'ltr', 'katakana': 'ltr', 'khmer': 'ltr', 'latin': 'ltr', 'mixed': 'ltr', 'mongolian': 'ttb', 'myanmar': 'ltr', 'south_asian': 'ltr', 'southeast_asian': 'ltr', 'symbolic': 'ltr', 'thai': 'ltr', 'tibetan': 'ltr', 'unknown': 'ltr'}#

SCRIPT_PROPERTY_MAP: ClassVar[dict[str, str]] = {'Arabic': 'arabic', 'Armenian': 'armenian', 'Common': '_common', 'Coptic': 'egyptian', 'Cyrillic': 'cyrillic', 'Devanagari': 'devanagari', 'Egyptian_Hieroglyphs': 'egyptian_hieroglyphs', 'Ethiopic': 'ethiopic', 'Georgian': 'georgian', 'Greek': 'greek', 'Han': 'han', 'Hangul': 'hangul', 'Hebrew': 'hebrew', 'Hiragana': 'hiragana', 'Inherited': '_inherited', 'Katakana': 'katakana', 'Khmer': 'khmer', 'Latin': 'latin', 'Mongolian': 'mongolian', 'Myanmar': 'myanmar', 'Thai': 'thai', 'Tibetan': 'tibetan'}#

segment(text)[source]#

Segment text into contiguous ScriptSpan objects.

Parameters:

textstr: NFC-normalised input text. Run GraphemeClusterNormalizer first for correctness on non-Latin scripts.

Returns:

list[ScriptSpan]: Non-empty list of spans in logical order. Returns a single ScriptType.UNKNOWN span for empty input.

Parameters:

text (str)

Return type:

list[ScriptSpan]

Notes

Developer note: Common / Inherited codepoints (punctuation, diacritics) are attributed to the preceding span; if there is no preceding span they are deferred and attributed to the next span. Whitespace-only spans are attributed to _common and merged into adjacent spans.