Skip to content
jsonW Paper GitHub

Lexicon

Lexicon: a text classifier with no training

A table of terms becomes an Aho-Corasick automaton that finds every term in a text in one pass, ignoring accents and letter case.

· jsonW 1.3.11

The problem

Sorting messages by intent, such as purchase, cancellation or complaint, usually takes a trained model or a list of regular expressions that grows out of control.

How jsonW solves it

A table with the schema termo:texto,forma:dic,sinal:dic,peso:f32,idioma:dic,origem:dic is compiled, on open, into an Aho-Corasick automaton. It matches whole words, ignoring accents and case, returns the original span and sums the weights per signal.

$ wjsw reconhecer termos.jsw "Quero CANCELAR o pedido, ou então me diga quanto custa o orcamento novo"
3 termo(s), 2 sinal(is)
  intenção de compra soma 2.2  vezes 2
  risco de perda       soma 2  vezes 1
  [6..14] CANCELAR  (termo 0, risco de perda, peso 2)
  [43..55] quanto custa  (termo 3, intenção de compra, peso 1.2)
  [58..67] orcamento  (termo 2, intenção de compra, peso 1)

Real output, in Portuguese. "CANCELAR" matched "cancelar" and "orcamento" matched "orçamento".

Where it is

wjsw reconhecer and the SDKs' Reconhecedor. It is useful for triage and for routing a request to the right specialist. It finds terms; understanding the sentence is still your model's job.

Read the full section in the paper →