register_tokenizer#

scikitplot.corpus.register_tokenizer(name, tokenizer)[source]#

Register a named TokenizerProtocol implementation.

Parameters:
namestr

Registry key.

tokenizerTokenizerProtocol

Tokenizer instance.

Parameters:
Return type:

None

Examples

>>> register_tokenizer("jieba", FunctionTokenizer(lambda t: t.split()))
>>> get_tokenizer("jieba").tokenize("hello world")
['hello', 'world']