Skip to content
jsonW Paper GitHub

Spelling, signals and block columns

Columns by block and by tenant: another customer's block is never opened

A local dictionary per block, fixed-width digits and the tenant in each block's zone. On the INPI dataset, the in-force count dropped from 54 to 43 ms.

· jsonW 1.3.11

The problem

In a file with data from many customers, one customer's query should not pay for reading the others. And codes made only of digits, such as a case number, take more space than they need when stored as text.

How jsonW solves it

Version 1.3.11 brought three codecs: a local dictionary per block, with and without entropy coding, and fixed-width digits. Each block's zone also stores the tenant. The filter compares dictionary codes in batches with SSE2 instructions and evaluates "contains" over the whole block at once.

$ wjsw calcular DADOS_BIBLIOGRAFICOS.jsw --onde "codigo_situacao = IPAS023"
1 grupo(s) em 44.0 ms
total: 2343607 linha(s)
$ wjsw calcular DESPACHOS.jsw --onde "numero_inpi = 818602856"
1 grupo(s) em 277.1 ms
total: 10 linha(s)
$ wjsw calcular CLASSIFICACOES_NICE.jsw --onde "classe_nice = 35"
1 grupo(s) em 145.3 ms
total: 1076928 linha(s)

Real output of the three Table 5D queries on the full INPI (Brazilian patent and trademark office) trademark dataset, 55.3 million rows, written by 1.3.11 at maximum level, in one run with the file cached. All three answers are correct. The paper uses the best of 5 runs: 43, 207 and 142 ms.

Where it is

When writing (the codecs apply automatically) and in filtrar and calcular. On the INPI dataset, with 8 cores, the in-force count took 43 ms, ahead of Parquet read by DuckDB (54 ms), DuckDB reading the CSV (375 ms) and grep on the CSV (519 ms). The case number column shrank from 53.0 to 25.7 MB. A file written with the new codecs does not open in 1.2.x readers.

Read the full section in the paper →