Skip to content
jsonW Paper GitHub

File layout

Zones: search skips entire blocks

Each block of the file stores the minimum and maximum of each column. In a 100,000-record file, a filter skipped 102 of 103 blocks.

· jsonW 1.3.11

The problem

In JSONL or CSV, to find the rows that pass a filter, the reader has to open and split every line, even if the answer is only at the end of the file.

How jsonW solves it

The .jsw file is written in blocks, and each block stores the minimum and maximum of each column. If the filter cannot be true in a block, the reader skips that block without decompressing any of it.

$ wjsw filtrar com.jsw "id > 99990"
9 de 100000 linhas em 1.7 ms · blocos 103, pulados pela zona 102 · linhas lidas 833
$ wjsw info prod.jsw
…
zona      preco        890 .. 2400
zona      nota         3.8999999999999999 .. 4.9000000000000004

Real output, on the 100,000-record file from Section 5 of the paper: 833 rows read to find 9. The info command shows the zones of another file, with reals in double precision.

Where it is

filtrar and calcular use the zones automatically, and info shows the values. The gain depends on data order: a column written in order skips many blocks, a shuffled column skips few.

Read the full section in the paper →