MultilangChunkMeta#

class scikitplot.corpus.MultilangChunkMeta(script=None, script_direction=None, is_mixed_script=None, chunking_unit=None, grapheme_count=None, codepoint_count=None, semantemes=None, semanteme_count=None, morphemes=None, script_spans=None, script_model_version=None, preprocessing_trace=None, raw_text=None, embedding=None, language_hint=None, model_name=None, chunking_duration_ms=None, preprocessing_duration_ms=None, layer2_strategy=None, pipeline_id=None, created_at_utc=None, char_offset_start=None, char_offset_end=None, token_count=None, stopword_count=None, unique_token_count=None, char_count=None, avg_token_length=None, is_rtl=None)[source]#

Per-chunk multilang analysis bundle — stored in chunk.metadata.

Collects all Layer 0-3 artefacts for a single chunk in one place so downstream consumers have a uniform access pattern regardless of which chunker produced the chunk.

Parameters:
scriptstr or None

Dominant ScriptType value string for this chunk.

script_directionstr or None

Writing direction: "ltr", "rtl", "ttb".

is_mixed_scriptbool or None

True if the chunk contains codepoints from more than one script.

chunking_unitstr or None

Granularity: "word", "sentence", "paragraph", "fixed_window", "grapheme_cluster", "semanteme".

grapheme_countint or None

UAX #29 grapheme cluster count for chunk.text.

codepoint_countint or None

len(chunk.text) — stored explicitly for comparison.

semantemeslist[SemantemeInfo] or None

Per-semanteme analysis list. None when semantic analysis was not applied.

semanteme_countint or None

len(semantemes) pre-computed for fast access.

morphemeslist[str] or None

Flat morpheme surface list (deduplicated from semantemes).

script_spanslist[dict] or None

ScriptSpan dicts for mixed-script chunks.

script_model_versionstr or None

Embedding model version used, format "name@version".

preprocessing_tracePreprocessingTrace or None

Full audit trail of preprocessing steps. None when include_preprocessing_trace=False.

raw_textstr or None

Original pre-processing text for this chunk’s span. None when include_raw_text=False.

embeddinglist[float] or None

Dense chunk-level embedding vector. None until an embedder is applied. Use with_embedding to attach one.

language_hintstr or None

ISO 639-1 / BCP-47 detected language code.

model_namestr or None

Name of the embedding model (e.g. "paraphrase-multilingual-mpnet-base-v2").

Parameters:

  • script (str | None)

  • script_direction (str | None)

  • is_mixed_script (bool | None)

  • chunking_unit (str | None)

  • grapheme_count (int | None)

  • codepoint_count (int | None)

  • semantemes (list[SemantemeInfo] | None)

  • semanteme_count (int | None)

  • morphemes (list[str] | None)

  • script_spans (list[dict[str, Any]] | None)

  • script_model_version (str | None)

  • preprocessing_trace (PreprocessingTrace | None)

  • raw_text (str | None)

  • embedding (list[float] | None)

  • language_hint (str | None)

  • model_name (str | None)

  • chunking_duration_ms (float | None)

  • preprocessing_duration_ms (float | None)

  • layer2_strategy (str | None)

  • pipeline_id (str | None)

  • created_at_utc (str | None)

  • char_offset_start (int | None)

  • char_offset_end (int | None)

  • token_count (int | None)

  • stopword_count (int | None)

  • unique_token_count (int | None)

  • char_count (int | None)

  • avg_token_length (float | None)

  • is_rtl (bool | None)

Notes

User note: Access via chunk.metadata.get("multilang") — it is stored as a dict (via to_dict). If you need to attach an embedding later, call with_embedding which returns a new frozen instance without mutating the original.

Developer note: All chunkers populate this via _build_multilang_meta. The mixin’s _build_multilang_meta is the single authority for constructing these objects — do not construct them inline in chunker code.

avg_token_length: float | None = None#

Average token length in grapheme clusters.

char_count: int | None = None#

Character count of chunk.text (len(chunk.text)).

char_offset_end: int | None = None#

End character offset of this chunk in the raw (pre-NFC) text.

char_offset_start: int | None = None#

Start character offset of this chunk in the raw (pre-NFC) text. Allows exact re-location of the original source span for comparison.

chunking_duration_ms: float | None = None#

Wall-clock time in milliseconds to produce this chunk. Populated by the chunker’s MultilangMixin._ml_build_meta.

chunking_unit: str | None = None#
codepoint_count: int | None = None#
created_at_utc: str | None = None#

ISO-8601 UTC timestamp of when this chunk was produced, format YYYY-MM-DDTHH:MM:SS.ffffffZ. None when not tracked.

embedding: list[float] | None = None#
grapheme_count: int | None = None#
is_mixed_script: bool | None = None#
is_rtl: bool | None = None#

True when script_direction == "rtl"; precomputed for fast dataframe filtering.

language_hint: str | None = None#
layer2_strategy: str | None = None#

Name of the Layer 2 SegmentationStrategy class that produced this chunk (e.g. "JapaneseStrategy", "ArabicMorphologicalStrategy", "GraphemeClusterStrategy").

model_name: str | None = None#
morphemes: list[str] | None = None#
pipeline_id: str | None = None#

Optional pipeline run identifier injected by the caller for correlation across large batch runs.

preprocessing_duration_ms: float | None = None#

Wall-clock time in milliseconds for all preprocessing steps combined (BOM strip + control strip + NFC normalisation).

preprocessing_trace: PreprocessingTrace | None = None#
raw_text: str | None = None#
script: str | None = None#
script_direction: str | None = None#
script_model_version: str | None = None#
script_spans: list[dict[str, Any]] | None = None#
semanteme_count: int | None = None#
semantemes: list[SemantemeInfo] | None = None#
stopword_count: int | None = None#

Number of stopwords among the tokens (if stopword analysis was run).

to_dict(*, include_raw_text=False, include_preprocessing_trace=False, include_embedding=False, include_semanteme_detail=True)[source]#

Serialise to a JSON-safe dict for chunk.metadata["multilang"].

Parameters:
include_raw_textbool

Include raw_text in output. Default False.

include_preprocessing_tracebool

Include the full preprocessing trace. Default False.

include_embeddingbool

Include the dense embedding vector. Default False.

include_semanteme_detailbool

Include full semantemes list. Default True. Set to False to include only semanteme_count.

Returns:
MetadataDict

All non-None fields populated per the flag settings.

Parameters:

  • include_raw_text (bool)

  • include_preprocessing_trace (bool)

  • include_embedding (bool)

  • include_semanteme_detail (bool)

Return type:

dict[str, Any]

token_count: int | None = None#

Number of whitespace tokens in chunk.text after normalisation.

unique_token_count: int | None = None#

Number of unique tokens (type count, not token count).

with_embedding(vector, *, model_name=None, model_version=None)[source]#

Return a copy with an embedding vector and optional model metadata.

Parameters:
vectorlist[float]

Dense embedding produced by any encoder.

model_namestr, optional

Name of the model used (e.g. "paraphrase-multilingual-mpnet-base-v2").

model_versionstr, optional

Model version string (e.g. "1.2.0").

Returns:
MultilangChunkMeta

New frozen instance.

Parameters:
Return type:

MultilangChunkMeta

Notes

CRITICAL-04 fix: uses dataclasses.replace so all fields not explicitly named here are propagated automatically. Any new fields added to MultilangChunkMeta will never be silently dropped.