SemanticChunkerConfig#

class scikitplot.corpus.SemanticChunkerConfig(backend=SemanticBackend.HYBRID, model_name='paraphrase-multilingual-mpnet-base-v2', model_version=None, similarity_threshold=0.85, max_chunk_tokens=None, min_chunk_tokens=1, include_offsets=True, adapter_unit='word', multilang_config=None, device='cpu')[source]#

Configuration for SemanticChunker.

Parameters:
backendSemanticBackend

Chunking backend. Default SemanticBackend.HYBRID.

model_namestr

Sentence-transformers model name used for EMBEDDING / HYBRID backends. Default "paraphrase-multilingual-mpnet-base-v2" (supports 50+ languages, 768-dim).

model_versionstr or None

Model version tag for script_model_version provenance field. Default None (auto-populated from installed package version when available).

similarity_thresholdfloat

Cosine similarity threshold for EMBEDDING / HYBRID boundary merging. Adjacent chunks with similarity >= this value are merged. Range [0.0, 1.0]. Default 0.85.

max_chunk_tokensint or None

Hard upper bound on tokens per semantic chunk. Chunks exceeding this are forcibly split by the Layer 2 adapter. Default None (no limit).

min_chunk_tokensint

Minimum token count. Chunks below this are merged into the adjacent larger chunk. Default 1.

include_offsetsbool

Compute and store character offsets. Default True.

adapter_unitstr

Layer 2 segmentation granularity: "word", "sentence", "grapheme_cluster". Default "word".

multilang_configMultilangConfig or None

Multilang feature flags for the mixin. Default None → MultilangConfig(include_semantemes=True).

devicestr

Torch device for embeddings: "cpu", "cuda", "mps". Default "cpu".

Parameters:

  • backend (SemanticBackend)

  • model_name (str)

  • model_version (str | None)

  • similarity_threshold (float)

  • max_chunk_tokens (int | None)

  • min_chunk_tokens (int)

  • include_offsets (bool)

  • adapter_unit (str)

  • multilang_config (Any)

  • device (str)

Notes

Developer note: model_name and model_version populate CorpusDocument.script_model_version via MultilangChunkMeta. This is the idempotency verification field — always set it when running EMBEDDING or HYBRID to detect model-version drift across pipeline re-runs.

adapter_unit: str = 'word'#
backend: SemanticBackend = 'hybrid'[source]#
device: str = 'cpu'#
include_offsets: bool = True#
max_chunk_tokens: int | None = None#
min_chunk_tokens: int = 1#
model_name: str = 'paraphrase-multilingual-mpnet-base-v2'#
model_version: str | None = None#
multilang_config: Any = None#
similarity_threshold: float = 0.85#