Newsroom
The 19 features of jsonW, one by one.
One post per feature of the format: the problem it solves, the command that shows the result, and what it does not do yet. The outputs are real, from the public binary or from the paper.
· jsonW 1.4.1
Search and ranking
Term search (IDF) inside the file
jsonW stores the sparse term vector with IDF weights in the row itself and scores it with the avaliar command, with no separate search engine.
02Semantic search with 32, 16, 8 and 4-bit vectors
Embeddings live in a contiguous vector column in the file, and avaliar computes the cosine against the query and returns the k closest rows.
03Multilinear grid: interpolation straight from the file
A table of values over axes, such as shipping cost by distance and weight, becomes a score through bilinear, trilinear or higher interpolation, inside the scoring request.
04Scoring by request: weights, cosine, sparse or grid, in one line
A one-line request scores every record in the file and returns the best ones, with the same result in the tool, the ABI and the 10 SDKs.
05The top k, always in the same order
jsonW's top-k uses a heap and is deterministic: highest score first and, on ties, the lower row. Two runs return the same list.
File layout
Zones: search skips entire blocks
Each block of the file stores the minimum and maximum of each column. In a 100,000-record file, a filter skipped 102 of 103 blocks.
07Dictionary: repeated text stored once
A column of type dic stores each distinct text once per file. City, status and category stop taking space on every row.
08Key index: straight to the row, no scan
The format has an index that leads to a key's row without reading the file. In the public tool, keys show up in the comparar command.
Lexicon
Writing and history
Live writing and atomic publishing
The .jsw file is append-only: one process can read while another writes, without seeing half a block. In the SDKs, the file only appears, complete, on fechar().
11Live, deleted and history in an append-only file
Writing the same key again edits it, writing the key with _apagado: true deletes it, and the history of every change stays in the file.
Permissions, custody and remote access
Permission views: column, row and cell
Permissions are declared in the schema and applied by whoever serves the file, before answering. Without the scope, the column comes out null; without the tenant, the row does not come out.
13Custody: chained CRC32C and SHA-256 on every block
Each block carries a CRC32C and a SHA-256 (or BLAKE3) hash, chained across the whole file. A single changed byte fails conferir.
14Partial remote reading: downloading 0.98% of a 79.1 MB file
With a header, a footer and HTTP Range requests, the reader downloads only the blocks of the rows it reads. It works with any web server, R2 and S3.
Spelling, signals and block columns
Similar spelling and sound: "Kokakola" finds "Coca-Cola"
The filter's ~~ operator gives each value a score from 0 to 1 by spelling and by a phonetic key, in Brazilian Portuguese, American English, Spanish and Chinese.
16Audio signals in one jsonW row
From 16-bit PCM, jsonW measures pitch, pauses, intensity, jitter, shimmer, HNR, formants, MFCC, spectral centroid, articulation rate and breaths.
17Text signals in Portuguese, English, Spanish and Chinese
Intensifiers, hedges, polarity, negation, passive voice, pronouns, all caps, punctuation, emoji, laughter and signs of irony, measured in one row.
18Video in scenes, with measurements for each segment
Frames and sound go in; out come the scene cuts, light, contrast, motion, colors, shape, dHash, the audio signals of each segment and a mosaic.
19Columns by block and by tenant: another customer's block is never opened
A local dictionary per block, fixed-width digits and the tenant in each block's zone. On the INPI dataset, the in-force count dropped from 54 to 43 ms.
Versions and security
jsonW 1.4.0: encryption at rest, document masks and agent protocol profiles
A .jsw file can stay encrypted on disk, with the key kept outside it. CPF and CNPJ take a mask that refuses an invalid value. And 49 objects of the agent protocols have a profile proven with a round trip.
02jsonW 1.4.1: the permission view applies to every command that reads values
In 1.4.0, only ver and the web server hid personal columns. 1.4.1 fixes it in every command that reads values: on the same test file, 23 of 37 checks failed before and 0 of 37 fail now.