StructureChunker#

class scikitplot.corpus.StructureChunker(config=None)[source]#

Chunk Markdown or reStructuredText by its headings and code blocks.

Parameters:

configStructureChunkerConfig or None, default=None: None is StructureChunkerConfig() (Markdown).

Parameters:

config (StructureChunkerConfig | None)

See also

RecursiveChunker: Splits prose that exceeds chunk_size.

Notes

User. Every chunk’s metadata holds heading_path (a tuple of the titles of the sections that contain it, outermost first), block ("prose", "code", "heading" or "front_matter") and section_level; documents built from the chunks carry all three. With include_heading, a heading directly followed by prose is in the text of that prose’s first chunk; any other heading is a chunk of its own, so no heading’s words are lost (unless min_length drops them). Code blocks are never merged with prose, and a code chunk starts at the beginning of its first line, so its indentation is kept.

Examples

>>> text = "# Guide\n\nIntro text.\n\n## Install\n\n```\npip install x\n```\n"
>>> chunks = StructureChunker().chunk(text).chunks
>>> [
...     (c.metadata["block"], c.metadata["heading_path"], c.text) for c in chunks
... ]
[('prose', ('Guide',), '# Guide\n\nIntro text.'),
 ('heading', ('Guide', 'Install'), '## Install'),
 ('code', ('Guide', 'Install'), '```\npip install x\n```')]

blocks(text)[source]#

Return the document’s blocks: (kind, span, level, title).

Parameters:

textstr: The source.

Returns:

list of (str, (int, int), int, str): kind is "heading" (with its level and title), "code", "prose" or "front_matter"; spans in source order, without blank lines between blocks.

Parameters:

text (str)

Return type:

list[tuple[str, tuple[int, int], int, str]]

chunk(text, doc_id=None, extra_metadata=None)[source]#

Split text into prose and code chunks with their heading path.

Parameters:

textstr: Raw document text.
doc_idstr or None, default=None: Stored in each chunk’s metadata.
extra_metadatadict or None, default=None: Merged into the result metadata.

Returns:

ChunkResult: Chunks in source order.

Raises:

TypeError: If text is not a str.
ValueError: If text is empty or whitespace-only, or holds a NUL character or a lone surrogate.

Parameters:

text (str)
doc_id (str | None)
extra_metadata (dict[str, Any] | None)

Return type:

ChunkResult

chunk_batch(texts, doc_ids=None, extra_metadata=None)[source]#

Chunk a list of documents.

Parameters:

textslist of str: Input documents.
doc_idslist of str or None, default=None: Parallel document identifiers.
extra_metadatadict or None, default=None: Shared metadata for every result.

Returns:

list of ChunkResult: One result per document.

Raises:

TypeError: If texts is not a list.
ValueError: If doc_ids length mismatches texts.

Parameters:

texts (list[str])
doc_ids (list[str] | None)
extra_metadata (dict[str, Any] | None)

Return type:

list[ChunkResult]

property config: StructureChunkerConfig#: The configuration (read-only).

config_class[source]#: alias of StructureChunkerConfig

strategy: ClassVar[ChunkingStrategy] = 'block'[source]#

text_is_source_slice = True#