PreprocessingTrace#

class scikitplot.corpus.PreprocessingTrace(raw_text, steps, final_text, pipeline_fingerprint, total_char_delta)[source]#

Ordered audit trail of all preprocessing transformations.

Provides full retrospective visibility into how raw text was transformed into the normalised text stored in text.

Parameters:

raw_textstr or None: The original raw text before any preprocessing. None when include_raw_text=False (the default, to save memory on large corpora).
stepslist[PreprocessingStep]: Ordered list of transformations, from first applied to last.
final_textstr: The normalised text after all steps.
pipeline_fingerprintstr: MD5 hex digest of step names (",".join(s.name for s in steps)). Use this to detect preprocessing pipeline changes across corpus versions or pipeline runs.
total_char_deltaint: Sum of all PreprocessingStep.char_delta values.

Parameters:

raw_text (str | None)
steps (list[PreprocessingStep])
final_text (str)
pipeline_fingerprint (str)
total_char_delta (int)

Notes

User note: Use PreprocessingTrace to:

Verify that preprocessing was applied consistently across corpus versions (compare pipeline_fingerprint).
Debug unexpected chunk boundaries by comparing raw_text vs final_text.
Feed the raw text into a separate embedding index for raw-vs-normalised retrieval comparison.

Developer note: PreprocessingTrace is stored in chunk.metadata["preprocessing_trace"] as a dict (via to_dict). It is NOT stored in CorpusDocument fields directly (too large); consumers must pull it from the chunk metadata.

Examples

>>> step = PreprocessingStep(
...     name="nfc_normalise", description="Apply NFC", changed=False, char_delta=0
... )
>>> trace = PreprocessingTrace.build(
...     raw_text="café", steps=[step], final_text="café"
... )
>>> trace.pipeline_fingerprint
'...'