modbot.training.vectorizers.TokVectorizer

class modbot.training.vectorizers.TokVectorizer(tokenizer=None, max_len=None)

Bases: object

Tokenizer vectorizer for LSTM primarily

Methods

encode(Y)	Encode target labels
fit(X)	Fit tokenizer on texts
fit_transform(X)	Fit transform method to conform to sklearn model format
load(inpath)	Load tokenizer
save(outpath)	Save tokenizer
tokenize(X)	Tokenize texts
transform(X)	Transform texts

Attributes

MAX_NB_WORDS
MAX_SEQUENCE_LENGTH

encode(Y)

Encode target labels

fit(X)

Fit tokenizer on texts

fit_transform(X)

Fit transform method to conform to sklearn model format

classmethod load(inpath)

Load tokenizer

save(outpath)

Save tokenizer

tokenize(X)

Tokenize texts

transform(X)

Transform texts