Skip to content
jsonW Paper GitHub

File layout

Dictionary: repeated text stored once

A column of type dic stores each distinct text once per file. City, status and category stop taking space on every row.

· jsonW 1.3.11

The problem

In JSON and CSV, a field like "status" or "city" repeats the same text over millions of rows. That costs disk, memory and read time.

How jsonW solves it

A column declared as dic stores each distinct text once and, on each row, only a short code. When querying, the filter looks the text up once in the dictionary and then compares codes.

$ wjsw de-texto "id:i64,produto:texto,categoria:dic,preco:decimal[2]" prod.jsonl prod.jsw
5 linhas -> prod.jsw
$ wjsw info prod.jsw
…
dicionario 2 textos

Real output: the category column of 5 products has 2 distinct texts. In the paper's 100,000 records, the tipo and fonte columns add up to 10.

Where it is

It is a schema type (categoria:dic), and info shows how many texts the dictionary holds. In version 1.3.11, feature 19 added a local dictionary per block, for text that repeats inside a block but not across the whole file.

Read the full section in the paper →