Cleaner#

class scikitplot.cleanprompt.Cleaner(plan=None, limits=None, prior=(), chunk_chars=None)[source]#

Apply one validated plan to text, files, folders and archives.

Parameters:

planCleanPlan, optional

Defaults to CleanPlan(): packs chosen by each file’s format, every format, placeholders.

limitsOfficeLimits, optional

Size bounds for a file read from disk (max_part_bytes), and for the members and total size of an Office file or archive.

prioriterable of Entry, optional

Entries issued earlier — a vault being appended to. A value among them keeps its stand-in, and new stand-ins never repeat one of theirs.

chunk_charsint, optional

Largest piece of a CSV, TSV or JSON Lines file encoded at once. Defaults to the policy’s max_input_chars. Larger record files are encoded piece by piece, cut between records, with the same result as one pass; any other format above the limit is refused.

Raises:

CleanPromptError

If the plan does not validate, or an entity engine it asks for is not installed.

Parameters:

• plan (CleanPlan | None)

• limits (OfficeLimits | None)

• prior (Iterable[Entry])

• chunk_chars (int | None)

Notes

User notes. Use it as a context manager, or call clear when done: the cleaner holds every value it removed until then. Its representation shows counts, never values.

Developer notes. Not thread-safe: the shared vault is the point of a cleaner, and two threads appending to it would number values in an order that depends on scheduling. Use one cleaner per thread when determinism across runs matters, which it does for anything tested.

Examples

Try it in your browser!

>>> from scikitplot.cleanprompt._plan import FluentCleanPrompt
>>> cleaner = FluentCleanPrompt().packs("patient").materialize()
>>> out = cleaner.encode_text('{"mrn": 20417, "email": "ann@example.com"}', "json")
>>> out.text
'{"mrn": -9900000001, "email": "[EMAIL-1]"}'
>>> cleaner.decode(out.text)
'{"mrn": 20417, "email": "ann@example.com"}'

Go BackOpen In Tab

catalog#

!! processed by numpydoc !!

clear()[source]#

Drop every value this cleaner holds. It cannot be used afterwards.

Return type:

None

decode(text, strict=False)[source]#

Put values back into text.

Parameters:

textstr

A model’s reply, or an encoded file edited by one.

strictbool, default=False

Raise on a placeholder this cleaner never issued.

Returns:

str

The restored text.

Parameters:

• text (str)

• strict (bool)

Return type:

str

decode_archive(source, target)[source]#

Restore every member of an encoded zip into a new zip.

Parameters:

sourcepath-like

An archive written by encode_archive.

targetpath-like

The zip to write.

Returns:

list of Item

One per member.

Parameters:

• source (str | PathLike)

• target (str | PathLike)

Return type:

list[Item]

See also

restore_archive

The same, given a vault instead of a cleaner.

decode_report(text, strict=False, kinds=None)[source]#

Put values back into text and say what was and was not restored.

Parameters:

textstr

A model’s reply, or an encoded file edited by one.

strictbool, default=False

Raise on a placeholder this cleaner never issued.

kindsiterable of str, optional

Restore only values of these kinds; placeholders of other kinds are left as they are. None restores every kind.

Returns:

RestorationResult

With unknown and repaired filled in.

Parameters:

• text (str)

• strict (bool)

• kinds (Iterable[str] | None)

Return type:

RestorationResult

decode_tree(source, target)[source]#

Restore every UTF-8 file under a folder into a mirror folder.

Parameters:

sourcepath-like

Encoded files, possibly edited by a model.

targetpath-like

Where restored files are written; must not be inside source.

Yields:

Item

One per file, in sorted path order.

Parameters:

• source (str | PathLike)

• target (str | PathLike)

Return type:

Iterator[Item]

See also

restore_tree

The same, given a vault instead of a cleaner.

encode_archive(source, target)[source]#

Encode every readable member of a zip into a new zip.

Parameters:

sourcepath-like

The zip to read.

targetpath-like

The zip to write. Must differ from source.

Returns:

list of Item

One per member, in name order.

Raises:

CleanPromptError

If the archive is unreadable, has too many members, or decompresses past the total size limit.

Parameters:

• source (str | PathLike)

• target (str | PathLike)

Return type:

list[Item]

Notes

Developer notes. The output is deterministic — members sorted, timestamps fixed — so encoding the same archive twice with fresh cleaners gives the same bytes. It is written to a temporary name and moved into place, so a failure never leaves half an archive.

encode_bytes(data, name, format=None)[source]#

Encode a file’s bytes, choosing the format by name unless given.

Parameters:

databytes

The file’s content.

namestr

The file name; its extension selects the format.

formatstr, optional

Force a format by name or extension.

Returns:

Encoded

The safe text and a report.

Raises:

CleanPromptError

If no selected format reads it, it is not UTF-8, it does not parse as its format, or it is a PDF or zip (which need a file).

Parameters:

• data (bytes)

• name (str)

• format (str | None)

Return type:

Encoded

encode_file(path, format=None)[source]#

Encode one file from disk.

Parameters:

pathpath-like

The file.

formatstr, optional

Force a format by name or extension.

Returns:

Encoded

The safe text and a report. Nothing is written.

Raises:

CleanPromptError

If the file is missing, too large, or cannot be read safely.

Parameters:

• path (str | PathLike)

• format (str | None)

Return type:

Encoded

encode_files(paths)[source]#

Encode several files into the one shared vault, in the order given.

Parameters:

pathsiterable of path-like

The files.

Returns:

list of Encoded

One per path.

Raises:

CleanPromptError

On the first file that cannot be encoded.

Parameters:

paths (Iterable[str | PathLike])

Return type:

list[Encoded]

encode_text(text, format='text', name=None)[source]#

Encode a string read as one format.

Parameters:

textstr

The content.

formatstr, default=’text’

A format name or extension. Formats whose text must first be extracted from bytes (Office, PDF, zip) are refused here.

namestr, optional

A name for reports; defaults to <text>.

Returns:

Encoded

The safe text and a report.

Raises:

TypeError

If text is not a string.

CleanPromptError

If the format is not selected, needs bytes, or the text does not parse as it.

Parameters:

• text (str)

• format (str)

• name (str | None)

Return type:

Encoded

encode_tree(source, target)[source]#

Encode every readable file under a folder into a mirror folder.

Parameters:

sourcepath-like

The folder to read.

targetpath-like

Where encoded files are written, mirroring source. Created if missing; must not be inside source.

Yields:

Item

One per file or skipped folder, in sorted path order.

Raises:

CleanPromptError

If source is not a folder or target is inside it.

Parameters:

• source (str | PathLike)

• target (str | PathLike)

Return type:

Iterator[Item]

Notes

User notes. It is a generator: files are encoded as you iterate, so wrap it in list to run it all. A file that is skipped or refused is not written, and says why.

fingerprint()[source]#

Return the plan’s content fingerprint.

Returns:

str

As CleanPlan.fingerprint.

Return type:

str

property formats: tuple[str, ...]#

Names of the formats this cleaner reads.

Type:

Tuple of str

handle()[source]#

Return a Handle over every value removed.

Returns:

Handle

Usable with decode, and exportable with Handle.export for a vault file.

Return type:

Handle

kinds_in(text)[source]#

Return the kinds of held values text refers to, sorted.

Parameters:

textstr

Text a model wrote: a reply, or a tool call’s arguments.

Returns:

tuple of str

One kind per distinct held value named in text — by its placeholder or stand-in, exactly or as the model rewrote it. Nothing it returns is a value.

Parameters:

text (str)

Return type:

tuple[str, …]

Notes

Developer notes. Decided by the same restoration that decoding uses, so “would this text receive a value of kind K” and “does decoding put one there” cannot disagree (CP-073).

leaks(text)[source]#

Return the kinds of held values that still occur in text.

Parameters:

textstr

Text about to leave, usually an encoded output.

Returns:

tuple of str

One kind per distinct value found, sorted; empty when clean.

Parameters:

text (str)

Return type:

tuple[str, …]

Notes

Developer notes. An independent second check, not a repeat of the first: encoding decides spans with detectors and arbitration; this searches the finished text for the removed values themselves, as whole tokens. It checks exactly the kinds remember repeats, so with remember on it can only fire on a defect, and with it off it fires on every recurrence the rules did not catch. Nothing it returns is a value.

property plan: CleanPlan#

The plan this cleaner runs.

Type:

CleanPlan

property policy: RedactionPolicy#

The policy labels are issued under.

Type:

RedactionPolicy

survey_tree(source)[source]#

Report what encoding a folder would do, writing nothing.

Parameters:

sourcepath-like

The folder.

Returns:

list of Item

As encode_tree would yield, with output left empty.

Parameters:

source (str | PathLike)

Return type:

list[Item]

Notes

User notes. A dry run to read before sending a folder anywhere: which files would be encoded, skipped or refused, and how many values of each kind each holds. No file is written and this cleaner’s vault is unchanged — the survey runs on a scratch copy of the plan.

text_format(format)[source]#

Return the name of the text format format resolves to here.

Parameters:

formatstr

A format name or extension.

Returns:

str

The format’s name.

Raises:

CleanPromptError

If the plan does not select it, or it is read from bytes.

Parameters:

format (str)

Return type:

str

vault()[source]#

Return a vault of every value removed so far.

Returns:

Vault

A fresh copy; clearing it does not clear the cleaner.

Return type:

Vault