RecursiveChunkerConfig#

class scikitplot.corpus.RecursiveChunkerConfig(chunk_size=1000, chunk_overlap=0, separators=('\n\n', '\n', '. ', ' ', ''), strip_whitespace=True, min_length=1)[source]#

Configuration for RecursiveChunker.

Parameters:

chunk_sizeint, default=1000

Largest chunk anchor, in characters, not counting the whitespace around it.

chunk_overlapint, default=0

Characters of the preceding text added as context (from a word start, never before the previous chunk’s start); 0 adds none. Must be smaller than chunk_size.

separatorstuple of str, default=(”nn”, “n”, “. “, “ “, “”)

Separators, coarsest first; must end with "" (characters). A list is accepted and stored as a tuple. The defaults assume "\n" line ends (readers normalise "\r\n"); add "\r\n\r\n" for text that keeps them. A hard-wrapped text is cut at line ends before sentence ends.

strip_whitespacebool, default=True

Narrow each chunk to exclude surrounding whitespace.

min_lengthint, default=1

Drop a chunk with fewer characters than this, not counting the whitespace around it (so a whitespace-only chunk is always dropped).

Parameters:

• chunk_size (int)

• chunk_overlap (int)

• separators (tuple[str, ...])

• strip_whitespace (bool)

• min_length (int)

chunk_overlap: int = 0#

chunk_size: int = 1000#

min_length: int = 1#

separators: tuple[str, ...] = ('\n\n', '\n', '. ', ' ', '')#

strip_whitespace: bool = True#