Skip to content
jsonW Paper GitHub

jsonW · json Wondrous

JSON was not built for AI.
jsonW was.

Today data goes through JSON, CSV, Parquet and a vector database. In jsonW, search, permissions and post-quantum encryption live in the file itself.

  • 6.6× smaller than JSONL
  • 2,310× faster to read a column
  • Every JSON is already jsonW

19 features

Four tools for the same data? One format.

Row, column, text and transport in one model, with search and permissions inside the file.

  1. Term searchsparse vector with IDF, the same math search engines use
  2. Semantic searchcosine over dense vf16 and vq8 vectors
  3. Multilinear gridN-linear interpolation: bilinear, trilinear and beyond
  4. Scoring requestweights, cosine, sparse or grid, in a one-line request
  5. Top-kheap-based top-k, always in the same order
  6. Zonesmin and max per block: searches skip what doesn't match
  7. Dictionaryrepeated text stored only once per file
  8. Key indexstraight to a key's row, no scan
  9. LexiconAho-Corasick automaton read straight from disk, accent- and case-insensitive
  10. Live writingread while another process writes, in real time
  11. Live, deleted and historyedit and delete by key in an append-only file
  12. Permission viewscolumn, row and cell, per credential
  13. Custodychained SHA-256 or BLAKE3: nothing changes unnoticed
  14. Partial remote readopens a file in the cloud and downloads only the blocks it reads
  15. Similar spelling and soundthe ~~ filter finds "Coca-Cola" when you search for "Kokakola", in Portuguese, English, Spanish and Chinese
  16. Audio signalspitch, pauses, intensity, jitter, shimmer, formants and speech rate, as one jsonW row
  17. Text signalsintensifiers, polarity, negation, passive voice, pronouns, laughter and irony cues, in four languages
  18. Video into scenesscene cuts, the signals of each stretch and a frame mosaic
  19. Per-block and per-tenant columnsdictionary per block and a tenant zone: another customer's block is never opened

Three forms

Row, column or text? The same data, the way each moment needs it.

The three forms are the same model. Going from one to another loses nothing and changes the type of no value.

Row

One value after another, in binary. For messages, events and real time.

Where JSON, MessagePack, CBOR or Protobuf are used today.

Column

The .jsw file: column blocks read straight from disk through mmap, with no copy.

Where JSONL, CSV or Parquet are used today.

Text

jsonW-text, for people to read and write. Every valid JSON is already valid jsonW-text.

Where JSON, TOML or YAML are used today.

# the same record, in jsonW-text (real output of wjsw ver)
{id:1,produto:"Café especial",preco:12.50 BRL,peso:250 g,sku:"CAF-0250",criado:t"2026-11-09T01:00:00.000000Z"}
{id:2,produto:"Pão de queijo",preco:8.90 BRL,peso:1000 g,sku:"PDQ-1000",criado:t"2026-11-09T01:05:00.000000Z"}

The input said "1 kg": the column is medida[g], so the value was converted to grams on write. Money keeps its currency, and time is exact to the microsecond.

flowchart LR
    M(("one data<br/>model"))
    M --- L["<b>Row</b><br/>binary, value after value<br/><i>messages · events · real time</i>"]
    M --- C["<b>Column</b><br/>.jsw file in blocks, read via mmap<br/><i>batch · history · training · search</i>"]
    M --- T["<b>Text</b><br/>jsonW-text, every JSON is valid<br/><i>people reading and writing</i>"]

Measured

How big and how fast? Smaller on disk, much faster to find a field.

6.6×smaller than
uncompressed JSONL
2,310×faster to read
a whole column
0dependencies: just C
and the standard library

Measured on 2026-11-13 with jsonW 1.3.11, on 100,000 records and 10 runs, against the same set in JSONL. Faster JSON readers narrow the gap.

With zstd or xz, JSONL is smaller than the .jsw at the default level. The methodology and the limits are in the paper. To measure on your machine, run wjsw medir.

The first page of the jsonW technical paper

Technical paper

Where is the study? In the jsonW paper.

  • The data model, the 19 features, security and custody, with executed examples.
  • Measurements redone on 2026-11-13, with the methodology, the recipe to repeat them and the limits. Includes the whole INPI trademark base against CSV, XML and Parquet.
  • Comparison with JSON, Parquet, Arrow, MessagePack, CBOR and Protobuf, including where jsonW loses.

Wortic Tecnologia · jsonW 1.3.11 · in six languages: Português · Español · Italiano · Français · 中文

Types

Which types does JSON lack? These.

Money, measures, exact decimals and codes come in the format, checked on write. Each column declares its type in a one-line schema.

Exact decimal

decimal[2] never goes through floating point. 12.50 stays 12.50.

Money

ISO 4217 currency on every value: 12.50 BRL, US$, €, ₿. Sums only within the same currency.

Measure

89 units in 14 quantities: length, mass, volume, energy, data, carbon and more. medida[kg] converts on write.

Comma or dot

The reader accepts 1.234,56 and 1,234.56. A JVM or PHP set to another locale does not swap the separator.

Code

codigo keeps the exact bytes: tax IDs, keys, plates and SKUs are never normalized or trimmed.

Full text

Complete Unicode, with accents, emoji and \0. +nfc normalizes when you ask.

Vector and tensor

vf32[768], vf16, bits, vq8 and tensor<bf16>[768,768], contiguous in the file.

Media

midia with hash, type, size and duration. The codec stays the open one: jsonW only carries it.

And the rest

Time in µs UTC, uuid, validated url, list, object, map, reference and qualquer for open JSON.

# the schema is one line of text. A new field comes in as null; a name never changes type.
id:i64,produto:texto,preco:dinheiro,peso:medida[g],sku:codigo,criado:tempo,cpf:codigo?+pessoal

The .jsw file

What if a write stops halfway? The file doesn't break.

Each block stands on its own. If a write is cut off midway, only the last block is lost, and wjsw reparar rebuilds the end.

Append-only

You can read while another process writes, like a tail -f that never sees half a block.

Direct access

Reads (row, field) without decoding the rest. Parallel reads per block and mmap across processes.

Skips what doesn't match

Each block keeps the min and max of each column, and searches skip whole blocks.

Custody

Chained crc32c and SHA-256 (or BLAKE3). Any changed data fails wjsw conferir.

Edit and delete

Writing the same key edits; _apagado: true deletes. Compaction rewrites only what's live.

Opens from afar

With the header and footer, over HTTP Range, the reader fetches only the blocks of the rows you read.

The wjsw tool

Already have JSONL? It becomes .jsw in one command.

And back to JSON, losing nothing. The same tool filters, sums, compares and verifies.

# JSON/JSONL to .jsw (--max: maximum compression)
wjsw de-texto "id:i64,produto:texto,preco:dinheiro,peso:medida[g]" dados.jsonl dados.jsw

wjsw ver dados.jsw                         # rows in jsonW-text
wjsw info dados.jsw                        # schema, blocks, size and hash
wjsw conferir dados.jsw                    # ok: 2 linhas, 1 blocos, sha256 e crc32c conferidos
wjsw para-texto dados.jsw --json           # back to JSON, lossless

wjsw filtrar dados.jsw "peso > 500"        # zones skip whole blocks
wjsw filtrar marcas.jsw "nome ~~en Kokakola" # similar spelling and sound
wjsw calcular dados.jsw --por produto --valor id
wjsw comparar ontem.jsw hoje.jsw id        # only in one, only in the other, changed, equal
wjsw reparar dados.jsw                     # cut-off file: rebuilds the end

Built for AI

Can you rank and classify inside the file? Yes, with vectors in it.

The pieces of an expert router and of any ranking over a .jsw, reading only the requested columns, in parallel.

Score

A score per row with weights, cosine over vectors, a sparse vector or an N-linear interpolation grid.

Pick

The top k, always in the same order: higher score first and, on a tie, the lower row.

Recognize

A table of terms becomes an Aho-Corasick automaton on open. Matches whole words, accent- and case-insensitive.

Signals

How someone spoke, wrote or appeared, measured from audio, text (pt, en, es, zh) and video, as rows in the declared schema.

wjsw avaliar dados.jsw pesos "preco=-1,nota=2" 10
wjsw avaliar dados.jsw cosseno vet "0.5,-1,2,0.25" 5
wjsw escolher dados.jsw preco 10
wjsw reconhecer termos.jsw "text to classify" --json

Built for what's next

What about quantum computing? jsonW is ready for it.

The data of an AI model, the vectors it searches and the answers it sends fit in the same format. The cryptography is the kind a quantum computer does not break.

Post-quantum transport

The jsonW wire runs over QUIC with TLS 1.3 and the post-quantum key exchange X25519MLKEM768, several streams in one connection, credit and resumption.

No key a quantum computer breaks

Custody and credentials use only 256-bit hashes and HMAC. Against a quantum computer, they keep 128 bits of security.

AI-generated media

bf16 and f16 tensors, progress while generating, and audio, image and video in chunks, with 0.04% envelope.

Security

Who sees what? What the session can't see never leaves the server.

Permissions live in the format itself, per column, per row and per cell.

Signed credential

HMAC-SHA256 checked in constant time, with nonce and sequence against replay, on top of TLS.

Permission views

+escopo(x) and +pessoal hide the column. With tenant, only the rows of the credential's customer appear.

Hardened reader

Bounds checked on every index, nesting capped at 1000 levels and fuzzing in the test suite.

sequenceDiagram
    autonumber
    participant App as Application
    participant S as Server (wjsw, wjsw-cgi or SDK)
    participant F as .jsw file
    App->>S: request + signed credential (HMAC-SHA256)
    S->>S: checks signature, expiry, nonce and sequence
    S->>F: opens only the credential tenant's blocks
    F-->>S: the customer's rows
    S->>S: hides +pessoal and +escopo(x) columns without the scope
    S-->>App: only what this session may see

Databases

Does it work with your database? Lossless both ways.

wcon takes data from the database to a .jsw and back, with the schema taken from each column's type. Values travel as data, never assembled into a command, and the way back is a single transaction.

flowchart LR
    PG[(PostgreSQL)] <--> W{{wcon}}
    MY[(MySQL · MariaDB)] <--> W
    SQ[(SQLite)] <--> W
    RD[(Redis · Valkey)] <--> W
    MG[(MongoDB)] <--> W
    W <--> J[".jsw file"]
    J -- "HTTP Range" --> R["wremoto<br/>R2 · S3 · HTTP"]

Exact numbers

numeric becomes an exact decimal and bigint stays a 64-bit integer. Nothing goes through floating point.

Vectors ready

pgvector's vector and halfvec become vf32 and vf16, ready for semantic search.

Really tested

PostgreSQL 14, 16 and 18 with the same md5 on both sides, MariaDB 11.8, SQLite, Redis 7 and MongoDB 7, under hostile load.

# Postgres to jsonW and back (the password stays in PGPASSWORD or ~/.pgpass)
WSQL_URL=postgresql://papel@host:5432/banco wcon pg "select * from vendas" vendas.jsw
wcon pg-gravar vendas.jsw public.vendas
wcon sqlite app.db "select * from clientes" clientes.jsw

Cloudflare R2 and S3

Is your data on Cloudflare R2? Query it right there.

The .jsw file stays in R2, Cloudflare's file storage, and is queried from there without downloading it all. wremoto reads the index, then only the parts that hold the answer. Since R2 doesn't charge for data egress, querying a 10 GB file costs only the piece read.

sequenceDiagram
    autonumber
    participant W as wremoto
    participant R as R2 / S3
    W->>R: Range: header and footer (schema, blocks, zones)
    R-->>W: a few KB
    W->>W: zones: which blocks can have "preco > 100"?
    W->>R: Range: only those blocks
    R-->>W: the requested blocks, nothing more
# each request is signed with AWS SigV4; the key stays in a file
export WJSW_R2_CRED=~/.cred/r2
wremoto info    r2://my-bucket/vendas.jsw
wremoto filtrar r2://my-bucket/vendas.jsw "preco > 100"

SDKs

Does it work in your language? The one you already use.

A stable C ABI and, on top of it, one SDK per language. Each one reads, writes and runs the operations, and all pass the same byte-for-byte test.

CC++17Node.jsPython 3PHP 8GoRustJava 22+KotlinSwift
# Node.js from npm: prebuilt for macOS and Linux, with types and the jsonw command
npm install @wortictecnologia/jsonw   # 1.4.1
import jsonw

with jsonw.Escritor("vendas.jsw", "id:i64,produto:texto,preco:dinheiro") as w:
    w.gravar({"id": 1, "produto": "Café especial", "preco": "12,50 BRL"})

with jsonw.Leitor("vendas.jsw") as r:
    print(r.campo(0, "preco"))   # direct access, without decoding the rest
    r.conferir()

License and transparency

How much does it cost? Free to use, clear about what it counts.

The jsonW Sustainable Use License allows use in your product, as a component. Without an agreement with Wortic, you may not offer jsonW itself, or a service that does what it does, to third parties. You may not reverse-engineer the binaries either.

The first time it runs on a machine, wjsw reports only the version and platform (v=1.4.1&so=linux-x86_64), so we can count installations. No name, path, data or IP is kept, and the report never repeats. To turn it off: JSONW_SEM_PING=1 or ~/.jsonw/sem-ping. The library and the SDKs never access the network.