File layout
Zones: search skips entire blocks
Each block of the file stores the minimum and maximum of each column. In a 100,000-record file, a filter skipped 102 of 103 blocks.
· jsonW 1.3.11
The problem
In JSONL or CSV, to find the rows that pass a filter, the reader has to open and split every line, even if the answer is only at the end of the file.
How jsonW solves it
The .jsw file is written in blocks, and each block stores the minimum and maximum of each column. If the filter cannot be true in a block, the reader skips that block without decompressing any of it.
$ wjsw filtrar com.jsw "id > 99990"
9 de 100000 linhas em 1.7 ms · blocos 103, pulados pela zona 102 · linhas lidas 833
$ wjsw info prod.jsw
…
zona preco 890 .. 2400
zona nota 3.8999999999999999 .. 4.9000000000000004
Real output, on the 100,000-record file from Section 5 of the paper: 833 rows read to find 9. The info command shows the zones of another file, with reals in double precision.
Where it is
filtrar and calcular use the zones automatically, and info shows the values. The gain depends on data order: a column written in order skips many blocks, a shuffled column skips few.