StructureChunkerConfig#

class scikitplot.corpus.StructureChunkerConfig(markup='markdown', chunk_size=1000, include_heading=True, separators=('\n\n', '\n', '. ', ' ', ''), min_length=1, front_matter=True)[source]#

Configuration for StructureChunker.

Parameters:

markup{“markdown”, “rst”}, default=”markdown”

The document’s markup; given, never guessed.

chunk_sizeint, default=1000

Largest chunk anchor, in characters, not counting the whitespace around it. Longer prose is split by RecursiveChunker’s rule; a longer code block at blank lines, then lines.

include_headingbool, default=True

When a heading is directly followed by prose, begin the context of that prose’s first chunk at the heading, so the heading’s words are part of the text that is retrieved. A heading that is not (it is followed by code, another heading or the end, or this is False) is a chunk of its own, block="heading".

separatorstuple of str, default=(”nn”, “n”, “. “, “ “, “”)

Separators for prose, coarsest first, ending with "".

min_lengthint, default=1

Drop a chunk with fewer characters than this, not counting the whitespace around it.

front_matterbool, default=True

Markdown only: a first line ---, followed by a line that is not blank and closed by a later line --- or ..., is YAML front matter (Pandoc’s rule), chunked like code with block="front_matter". Otherwise, or with False, the --- is read as CommonMark reads it (a thematic break).

Parameters:

• markup (str)

• chunk_size (int)

• include_heading (bool)

• separators (tuple[str, ...])

• min_length (int)

• front_matter (bool)

chunk_size: int = 1000#

front_matter: bool = True#

include_heading: bool = True#

markup: str = 'markdown'#

min_length: int = 1#

separators: tuple[str, ...] = ('\n\n', '\n', '. ', ' ', '')#