Downloads and network boundaries#

DownloadPolicy: one transport contract#

DownloadPolicy centralizes the settings that describe transport security and resource budgets. It contains no credentials, headers or output paths, so it can be serialized in configuration/evidence without accidentally serializing authentication material.

Built-in presets keep security controls enabled:

secure

General-purpose defaults: TLS verification, private-IP blocking, bounded redirects, retries and 100 MiB maximum transfer.

constrained / ci / docs

Smaller timeout/byte/retry budgets for deterministic CI and documentation. It does not disable TLS or SSRF protection.

large / large-files

Larger byte/time budget while preserving the same security boundaries.

from scikitplot.corpus import AnyDownloader, DownloadPolicy

policy = DownloadPolicy.from_config({
    "preset": "constrained",
    "max_bytes": 8 * 1024 * 1024,
})
downloader = AnyDownloader.from_policy(
    "https://example.com/data.csv",
    policy,
)

Unknown configuration keys and non-boolean security flags fail early. This prevents values such as verify_ssl="false" from being interpreted as a truthy value. Explicit keyword arguments may still override a policy when the caller intentionally needs a per-transfer exception.

Plan dispatch without touching the network#

Use AnyDownloader.plan when code, CI or a user interface needs to show what would happen before any transfer starts. The returned DownloadPlan uses the same URL-classification/per-URL parameter resolver as the real specialist construction. Planning intentionally does not resolve DNS, issue HEAD/GET requests, allocate a temporary directory or import a remote SDK.

from scikitplot.corpus import AnyDownloader, DownloadPolicy

downloader = AnyDownloader.from_policy(
    "https://github.com/org/repo/blob/main/data.csv",
    DownloadPolicy.constrained(),
    github_token="...",
)
plan = downloader.plan()
print(plan.downloader)              # GitHubDownloader
print(plan.max_bytes)               # constrained transfer budget
print(plan.github_token_configured) # True; token value is not serialized

DownloadPlan reports whether credentials/custom headers are configured but never includes their values. It is a dispatch plan, not a security verdict: SSRF DNS resolution and remote content-type probing belong to execution-time network boundaries and are intentionally not disguised as side-effect-free preflight. For batch inputs, AnyDownloader.plan_all always returns one plan per URL in input order.

Builder integration#

CorpusBuilder no longer owns a second direct-download implementation. Downloadable URLs flow through _make_downloader() and then through the same _make_reader() seam used by local files.

This separation matters because transfer policy and parsing policy are different:

URL -> downloader -> DownloadResult(output_path, suffix, MIME, filename)
                                 |
                                 v
                           DocumentReader

A temporary local filename is implementation detail; provenance can continue to point at the original URL.

Advanced built-in downloader tuning does not require a custom factory. Put explicit overrides in BuilderConfig downloader_kwargs:

from scikitplot.corpus import BuilderConfig, CorpusBuilder

builder = CorpusBuilder(BuilderConfig(
    download_policy="constrained",
    downloader_kwargs={
        "max_redirects": 2,
        "youtube_language": "tr",
    },
))

Call-site downloader kwargs take precedence over this mapping; the mapping takes precedence over ordinary builder defaults. Security-reducing values such as verify_ssl=False or block_private_ips=False are therefore explicit user choices rather than hidden fallbacks.

Security defaults#

The built-in downloader layer keeps SSRF, redirect, scheme, TLS and size controls. CustomDownloader is intentionally a trusted-code escape hatch: its handler can execute arbitrary Python and could make unrelated network requests, so the wrapper cannot sandbox it. Instead the wrapper applies what it can verify independently:

• the original input URL still receives the normal SSRF pre-check by default;

• timeout, size, TLS, redirect and user-agent policy are passed to the handler;

• the returned path must exist and resolve to a regular file;

• it must remain inside the configured output directory unless allow_external_output=True is explicit;

• returned file size is checked against max_bytes by default even if the handler ignored the hint.

A handler remains responsible for applying those transfer-policy arguments to its own redirects/subrequests. Do not accept an untrusted downloaded Python callable as a CustomDownloader handler. If a private network or custom transport is intentionally allowed, make that decision in the custom downloader/policy rather than weakening the global default.

Runtime plans#

RuntimePolicy defaults to allow_network=False. A Fluent plan with a URL source therefore requires an explicit network permission at execution time. This is independent of backend/model download policy: allowing an input URL does not automatically mean an OCR/ASR/model backend may fetch assets.

Policy/legacy configuration ambiguity#

downloader_kwargs remains the deliberate per-build override layer.

Do not mix an explicit download_policy with customized legacy scalar fields such as download_timeout or max_download_bytes. That is ambiguous and BuilderConfig rejects it instead of silently ignoring one side. Put the value in DownloadPolicy, or use downloader_kwargs when an intentional one-build override is required.