Corpus User Guide#
scikitplot.corpus turns local files, URLs, archives and media into
canonical CorpusDocument evidence, then lets you normalize, enrich,
embed, store, retrieve and export that evidence without requiring one monolithic
runtime.
The important boundary is that configuration, capability discovery and execution are separate. A package can be installed without its model or system binary being ready; a backend can be selected without becoming active; and a fail-soft operation can continue while still recording structured degradation.
Important
Corpus treats optional capability failure as data, not as success. Use
component_capabilities before a run when readiness matters and inspect
reader.backend_reports after a reader run when a fallback was possible.
strict=False may keep ingestion alive; it does not mean failures are
invisible.
Read the first two pages first. After that, choose the page for the task in front of you.
evaluate
troubleshoot
- Troubleshooting
- Package installed, backend still unavailable
- Whisper fails but the gallery should continue
- Need a fully local ASR
- A builder factory seems ignored
- A URL works in a browser but Corpus refuses it
- A Levenshtein backend is missing
- Offline policy says unavailable, not empty
- A segmenter warning appears once, then never again
ResourceWarning: unclosed database
The executable gallery, Corpus, follows the same architecture with deterministic local examples first and optional live/model paths clearly marked.
The whole idea in one picture#
source
|
v
downloader / local path
|
v
DocumentReader ---- capability/readiness ---- optional backends
| \
| +-- fallback policy + reports
v
filter -> normalize -> chunk -> enrich -> embed -> store -> index
|
v
retrieve / adapt / export
There are three complementary configuration levels:
CorpusPipelineDirect control of one execution pipeline.
CorpusBuilderHigh-level heterogeneous ingestion, downloading, search and adaptation.
FluentCorpusImmutable declarative plans that can be generated, branched, compared and materialized explicitly.
Thirty seconds#
from scikitplot.corpus import FluentCorpus
plan = FluentCorpus.from_config({
"chunker": "paragraph",
"storage": "memory",
})
plan.explain()["configured"]
# ['chunker', 'storage']
Inspect optional component readiness without importing heavy models:
from scikitplot.corpus import component_capabilities
status = component_capabilities(["asr:faster-whisper", "ocr:pytesseract"])
status["asr:faster-whisper"]["installed"]
# True or False depending on the environment
Which surface should I use?#
Goal |
Start with |
|---|---|
Process one source with explicit stages |
|
Ingest heterogeneous local/remote sources |
|
Generate reusable configuration variants |
|
Supply a custom reader/filter/downloader/component |
|
Inspect optional package/model/binary readiness |
|
Preview the exact ASR chain without running models |
|
Control backend order/fallback/offline behavior |
|
Reuse one typed policy family across runtime/reader/builder seams |
|
Control transfer TLS/SSRF/size/retry budgets |
|
Add a private/local ASR implementation |
|
Download HTTP/GDrive/GitHub/YouTube inputs |
|
Do fuzzy lexical ranking without requiring Corpus |
|
Search vectors/lexical/hybrid indexes |
|
Export/adapt documents |
the export and adapter APIs (Formats, structured evidence, and export) |