Lexicon
Lexicon: a text classifier with no training
A table of terms becomes an Aho-Corasick automaton that finds every term in a text in one pass, ignoring accents and letter case.
· jsonW 1.3.11
The problem
Sorting messages by intent, such as purchase, cancellation or complaint, usually takes a trained model or a list of regular expressions that grows out of control.
How jsonW solves it
A table with the schema termo:texto,forma:dic,sinal:dic,peso:f32,idioma:dic,origem:dic is compiled, on open, into an Aho-Corasick automaton. It matches whole words, ignoring accents and case, returns the original span and sums the weights per signal.
$ wjsw reconhecer termos.jsw "Quero CANCELAR o pedido, ou então me diga quanto custa o orcamento novo"
3 termo(s), 2 sinal(is)
intenção de compra soma 2.2 vezes 2
risco de perda soma 2 vezes 1
[6..14] CANCELAR (termo 0, risco de perda, peso 2)
[43..55] quanto custa (termo 3, intenção de compra, peso 1.2)
[58..67] orcamento (termo 2, intenção de compra, peso 1)
Real output, in Portuguese. "CANCELAR" matched "cancelar" and "orcamento" matched "orçamento".
Where it is
wjsw reconhecer and the SDKs' Reconhecedor. It is useful for triage and for routing a request to the right specialist. It finds terms; understanding the sentence is still your model's job.