MultilangMixin#

class scikitplot.corpus.MultilangMixin[source]#

Shared multilang analysis mixin for all five chunkers.

Inherit this alongside the chunker base class:

class WordChunker(MultilangMixin, ChunkerBase): ...

The mixin adds no __init__ parameters. Configuration is passed via MultilangConfig held in self._ml_cfg. Subclasses must call _ml_init in their __init__ to set this up.

Notes

Developer note: All mixin methods are name-mangled with _ml_ to avoid collisions with subclass methods. The only external surface is attach_embedding and attach_embedding_batch.

Thread safety: All methods are stateless per-call. The only shared state is _REGEX_AVAILABLE (module-level bool, written once).

attach_embedding(chunk, vector, *, model_name=None, model_version=None)[source]#

Return a new Chunk with an embedding attached.

Does NOT mutate the original Chunk (frozen dataclass).

Parameters:
chunkChunk

Any chunk produced by this chunker.

vectorlist[float]

Dense embedding vector.

model_namestr, optional

Encoder model name.

model_versionstr, optional

Encoder model version.

Returns:
Chunk

New frozen instance with metadata["multilang"]["embedding"] populated and metadata["embedding"] set at top level for compatibility with EmbeddedChunk.

Parameters:
Return type:

Chunk

Notes

User note: For batch embedding, use attach_embedding_batch which avoids per-chunk dict copies.

Developer note: Two embedding locations are written:

  1. chunk.metadata["embedding"] — top-level key compatible with EmbeddedChunk and vector store adapters.

  2. chunk.metadata["multilang"]["embedding"] — inside the multilang bundle for model provenance tracking.

attach_embedding_batch(chunks, vectors, *, model_name=None, model_version=None)[source]#

Return a new list of chunks with embeddings attached.

Parameters:
chunkslist[Chunk]

Chunks from this chunker.

vectorslist[list[float]]

One embedding vector per chunk. Must have same length as chunks.

model_namestr, optional

Encoder model name.

model_versionstr, optional

Encoder model version.

Returns:
list[Chunk]

New list; originals are unmodified.

Raises:
ValueError

If len(chunks) != len(vectors).

Parameters:
Return type:

list[Chunk]