Corpus User Guide#

scikitplot.corpus turns local files, URLs, archives and media into canonical CorpusDocument evidence, then lets you normalize, enrich, embed, store, retrieve and export that evidence without requiring one monolithic runtime.

The important boundary is that configuration, capability discovery and execution are separate. A package can be installed without its model or system binary being ready; a backend can be selected without becoming active; and a fail-soft operation can continue while still recording structured degradation.

Important

Corpus treats optional capability failure as data, not as success. Use component_capabilities before a run when readiness matters and inspect reader.backend_reports after a reader run when a fallback was possible. strict=False may keep ingestion alive; it does not mean failures are invisible.

Read the first two pages first. After that, choose the page for the task in front of you.

The executable gallery, Corpus, follows the same architecture with deterministic local examples first and optional live/model paths clearly marked.

The whole idea in one picture#

source
   |
   v
downloader / local path
   |
   v
DocumentReader ---- capability/readiness ---- optional backends
   |                                      \
   |                                       +-- fallback policy + reports
   v
filter -> normalize -> chunk -> enrich -> embed -> store -> index
                                                  |
                                                  v
                                       retrieve / adapt / export

There are three complementary configuration levels:

CorpusPipeline

Direct control of one execution pipeline.

CorpusBuilder

High-level heterogeneous ingestion, downloading, search and adaptation.

FluentCorpus

Immutable declarative plans that can be generated, branched, compared and materialized explicitly.

Thirty seconds#

from scikitplot.corpus import FluentCorpus

plan = FluentCorpus.from_config({
    "chunker": "paragraph",
    "storage": "memory",
})
plan.explain()["configured"]
# ['chunker', 'storage']

Inspect optional component readiness without importing heavy models:

from scikitplot.corpus import component_capabilities

status = component_capabilities(["asr:faster-whisper", "ocr:pytesseract"])
status["asr:faster-whisper"]["installed"]
# True or False depending on the environment

Which surface should I use?#

Goal

Start with

Process one source with explicit stages

CorpusPipeline

Ingest heterogeneous local/remote sources

CorpusBuilder

Generate reusable configuration variants

FluentCorpus (Fluent configuration and policies)

Supply a custom reader/filter/downloader/component

CorpusBuilder with BuilderFactories (Customization)

Inspect optional package/model/binary readiness

component_capabilities (Readers, backends, and readiness)

Preview the exact ASR chain without running models

AudioReader.plan_asr_backends / BackendPlan

Control backend order/fallback/offline behavior

BackendPolicy

Reuse one typed policy family across runtime/reader/builder seams

CorpusPolicyBundle (Fluent configuration and policies)

Control transfer TLS/SSRF/size/retry budgets

DownloadPolicy (Downloads and network boundaries)

Add a private/local ASR implementation

ASRBackend

Download HTTP/GDrive/GitHub/YouTube inputs

AnyDownloader (Downloads and network boundaries)

Do fuzzy lexical ranking without requiring Corpus

scikitplot.levenshtein (Retrieval and similarity)

Search vectors/lexical/hybrid indexes

RetrievalIndex

Export/adapt documents

the export and adapter APIs (Formats, structured evidence, and export)