assign_doc_ids#

scikitplot.corpus.assign_doc_ids(documents, scheme=DocIdScheme.SHA256, *, generator=None)[source]#

Return documents named by scheme, with parent links kept intact.

Parameters:
documentsiterable of CorpusDocument

Documents in reading order. Each needs doc_id, parent_doc_id and replace(**changes).

schemeDocIdScheme or str, optional

"sha256" (default) keeps the content-addressed ids the producers derived. "uuid7" mints one identifier per document, in order, so doc_id order is the order of documents.

generatorcallable, optional

Returns a uuid.UUID; defaults to uuid7. For a caller that keeps its own UUID7Generator.

Returns:
list of CorpusDocument

New documents for "uuid7"; the same objects for "sha256".

Raises:
ValueError

For an unknown scheme. For "uuid7": if a parent_doc_id names an id that two documents of the batch share, so its new id would be ambiguous; if generator returns something other than a version-7 UUID, or an identifier not greater than the one before (a repeat or a step back would break uniqueness and order).

Parameters:
Return type:

list[Any]

Notes

User. The builder and the pipeline call this once per source, right after it is read and before normalisation, enrichment and embedding, so no stage sees two names for one document.

Developer. A parent link to a document of the batch follows it to its new id. A link to a document outside the batch is kept verbatim: it names a document stored elsewhere (a previous build, another source), which this call cannot rename, and scikitplot.corpus._hierarchy already treats such a parent as external. Nothing is minted until the whole batch has been read, so an iterable that fails part-way consumes no identifiers.

Examples

>>> from scikitplot.corpus import CorpusDocument
>>> parent = CorpusDocument.create("a.txt", 0, "Part one.")
>>> child = CorpusDocument.create("a.txt", 1, "Part one, section one.")
>>> child = child.replace(parent_doc_id=parent.doc_id)
>>> p, c = assign_doc_ids([parent, child], "uuid7")
>>> (c.parent_doc_id == p.doc_id, p.doc_id < c.doc_id)
(True, True)