SemanticChunker#

class scikitplot.corpus.SemanticChunker(config=None, *, adapter=None)[source]#

Embedding-aware, writing-system-aware semantic chunker (Layer 3).

Combines Layer 1 (ScriptSegmenter) and Layer 2 (WritingSystemAdapter) for linguistically correct segmentation across all Unicode scripts, then optionally refines boundaries with sentence-transformers embeddings.

Parameters:

configSemanticChunkerConfig, optional: Full configuration. Default: HYBRID backend, multilingual mpnet.
adapterWritingSystemAdapter or None, optional: Layer 2 adapter injected via Dependency Injection (Q5). When None, a default adapter is constructed from config.adapter_unit. Pass a custom adapter to swap per-script strategies without subclassing.

Parameters:

config (SemanticChunkerConfig | None)
adapter (WritingSystemAdapter | None)

Notes

User note: Install sentence-transformers and regex for full HYBRID functionality:

pip install sentence-transformers regex

For MORPHOLOGICAL-only mode (no embeddings, fully offline), set backend=SemanticBackend.MORPHOLOGICAL.

User note (raw text): Set multilang_config=MultilangConfig(include_raw_text=True, include_preprocessing_trace=True) to preserve the pre-NFC raw text alongside the normalised text in each chunk for comparison.

Developer note (Q5 — Dependency Injection): adapter is a constructor parameter, not an internal attribute. This makes SemanticChunker fully testable without touching the global strategy registry, and composable with custom strategies.

Developer note (embedding hook): Call attach_embedding_batch after chunking to add vectors to existing chunks without re-running the full pipeline.

Examples

>>> chunker = SemanticChunker()
>>> result = chunker.chunk("Hello world. مرحبا بالعالم。")
>>> result.chunks[0].metadata["multilang"]["script"]
'latin'

>>> # MORPHOLOGICAL only — no network, no GPU
>>> from ._semantic import SemanticBackend, SemanticChunkerConfig
>>> cfg = SemanticChunkerConfig(backend=SemanticBackend.MORPHOLOGICAL)
>>> chunker_morph = SemanticChunker(cfg)

>>> # Inject custom adapter
>>> from ._writing_system import WritingSystemAdapter, WritingSystemAdapterConfig
>>> adapter = WritingSystemAdapter(WritingSystemAdapterConfig(unit="sentence"))
>>> chunker_sent = SemanticChunker(adapter=adapter)

attach_embedding(chunk, vector, *, model_name=None, model_version=None)[source]#

Return a new Chunk with an embedding attached.

Does NOT mutate the original Chunk (frozen dataclass).

Parameters:

chunkChunk: Any chunk produced by this chunker.
vectorlist[float]: Dense embedding vector.
model_namestr, optional: Encoder model name.
model_versionstr, optional: Encoder model version.

Returns:

Chunk: New frozen instance with metadata["multilang"]["embedding"] populated and metadata["embedding"] set at top level for compatibility with EmbeddedChunk.

Parameters:

chunk (Chunk)
vector (list[float])
model_name (str | None)
model_version (str | None)

Return type:

Chunk

Notes

User note: For batch embedding, use attach_embedding_batch which avoids per-chunk dict copies.

Developer note: Two embedding locations are written:

chunk.metadata["embedding"] — top-level key compatible with EmbeddedChunk and vector store adapters.
chunk.metadata["multilang"]["embedding"] — inside the multilang bundle for model provenance tracking.

attach_embedding_batch(chunks, vectors, *, model_name=None, model_version=None)[source]#

Return a new list of chunks with embeddings attached.

Parameters:

chunkslist[Chunk]: Chunks from this chunker.
vectorslist[list[float]]: One embedding vector per chunk. Must have same length as chunks.
model_namestr, optional: Encoder model name.
model_versionstr, optional: Encoder model version.

Returns:

list[Chunk]: New list; originals are unmodified.

Raises:

ValueError: If len(chunks) != len(vectors).

Parameters:

chunks (list[Chunk])
vectors (list[list[float]])
model_name (str | None)
model_version (str | None)

Return type:

list[Chunk]

chunk(text, doc_id=None, extra_metadata=None)[source]#

Segment text using Layer 1 → Layer 2 → optional Layer 3.

Parameters:

textstr: NFC-normalised (or raw, will be normalised internally) text.
doc_idstr, optional: Document identifier stored in each chunk’s metadata.
extra_metadatadict[str, Any], optional: Extra key/value pairs merged into result metadata.

Returns:

ChunkResult: Ordered chunks with multilang metadata.

Raises:

TypeError: If text is not a str.
ValueError: If text is empty or whitespace-only, or contains NUL bytes or lone surrogates.

Parameters:

text (str)
doc_id (str | None)
extra_metadata (dict[str, Any] | None)

Return type:

ChunkResult

Notes

Algorithm:

Input validation (NUL / surrogate guard).
Multilang preprocessing trace (BOM strip → control strip → NFC).
Layer 1: ScriptSegmenter → ScriptSpan list.
Layer 2: WritingSystemAdapter → per-script Chunk list.
Min/max token filtering and enforcement.
EMBEDDING / HYBRID: boundary refinement via cosine similarity.
MultilangMixin: enrich each chunk with MultilangChunkMeta.

chunk_batch(texts, doc_ids=None, extra_metadata=None)[source]#

Chunk a list of documents.

Parameters:

textslist[str]: Input documents.
doc_idslist[str], optional: Parallel document identifiers.
extra_metadatadict[str, Any], optional: Shared metadata for every result.

Returns:

list[ChunkResult]: One result per document.

Raises:

TypeError: If texts is not a list.
ValueError: If doc_ids length mismatches texts.

Parameters:

texts (list[str])
doc_ids (list[str] | None)
extra_metadata (dict[str, Any] | None)

Return type:

list[ChunkResult]

Gallery examples#

Compare Corpus Chunking Strategies on OCR Text