Function: createTokenizer()

createTokenizer(model): Tokenizer

Create a tokenizer for a specific model

Parameters

Returns

Tokenizer

Source

src/model/utils/tokenizer.ts:10 (opens in a new tab)