RecursiveChunker#

class scikitplot.corpus.RecursiveChunker(config=None)[source]#

Split text by a separator hierarchy until each piece fits a budget.

Parameters:

configRecursiveChunkerConfig or None, default=None: None is RecursiveChunkerConfig().

Parameters:

config (RecursiveChunkerConfig | None)

See also

ParentChildChunker: The same rule at two sizes, linked.

Notes

User. A paragraph that fits stays whole; a longer one is cut at line ends, then sentence ends, then spaces, then characters, only as far as needed. Each chunk’s start_char/end_char is the piece it stands for; with chunk_overlap its text also holds the end of the previous chunk (context_span).

Examples

>>> text = "Alpha one. Beta two.\n\nGamma three."
>>> chunks = (
...     RecursiveChunker(RecursiveChunkerConfig(chunk_size=12)).chunk(text).chunks
... )
>>> [c.text for c in chunks]
['Alpha one.', 'Beta two.', 'Gamma three.']
>>> all(c.text == text[slice(*c.context_span)] for c in chunks)
True

chunk(text, doc_id=None, extra_metadata=None)[source]#

Split text into chunks of at most chunk_size characters.

Parameters:

textstr: Raw document text.
doc_idstr or None, default=None: Stored in each chunk’s metadata.
extra_metadatadict or None, default=None: Merged into the result metadata.

Returns:

ChunkResult: The chunks; metadata["span_basis"] is "input".

Raises:

TypeError: If text is not a str.
ValueError: If text is empty or whitespace-only, or holds a NUL character or a lone surrogate.

Parameters:

text (str)
doc_id (str | None)
extra_metadata (dict[str, Any] | None)

Return type:

ChunkResult

chunk_batch(texts, doc_ids=None, extra_metadata=None)[source]#

Chunk a list of documents.

Parameters:

textslist of str: Input documents.
doc_idslist of str or None, default=None: Parallel document identifiers.
extra_metadatadict or None, default=None: Shared metadata for every result.

Returns:

list of ChunkResult: One result per document.

Raises:

TypeError: If texts is not a list.
ValueError: If doc_ids length mismatches texts.

Parameters:

texts (list[str])
doc_ids (list[str] | None)
extra_metadata (dict[str, Any] | None)

Return type:

list[ChunkResult]

property config: RecursiveChunkerConfig#: The configuration (read-only).

config_class[source]#: alias of RecursiveChunkerConfig

spans(text)[source]#

Return the anchor span of every chunk chunk would produce.

Parameters:

textstr: The source.

Returns:

list of (int, int): Anchor spans in source order.

Raises:

TypeError, ValueError: As chunk.

Parameters:

text (str)

Return type:

list[tuple[int, int]]

strategy: ClassVar[ChunkingStrategy] = 'recursive'[source]#

text_is_source_slice = True#