Permissions, custody and remote access
Partial remote reading: downloading 0.98% of a 79.1 MB file
With a header, a footer and HTTP Range requests, the reader downloads only the blocks of the rows it reads. It works with any web server, R2 and S3.
· jsonW 1.3.11
The problem
To read one row of a file stored on a server or in a bucket, most formats force you to download the whole file first. That costs time and paid egress.
How jsonW solves it
The reader downloads the header and footer, which hold the schema, the block index and the zones, and then requests only the blocks it needs, via HTTP Range. Zones discard blocks before they are downloaded.
$ wremoto info http://127.0.0.1:18099/cartoes.jsw
… 100000 linhas em 103 blocos · arquivo 79.1 MB · abrir baixou 8.7 KB (cabecalho + rodape)
$ wremoto ver http://127.0.0.1:18099/cartoes.jsw 50000 1
{id:50000,tipo:"fator de emissão",titulo:"Fator de emissão de CO₂ do combustível nº 50000",…}
baixado no total: 777.8 KB em 5 pedidos (0.98% do arquivo)
$ wremoto filtrar http://127.0.0.1:18099/cartoes.jsw "id > 99990"
…
baixado no total: 667.7 KB em 5 pedidos (0.84% do arquivo)
Real wremoto output against a local nginx: opening downloaded 8.7 KB; reading one row from the middle, 0.98% of the file; the filter, 0.84%.
Where it is
wremoto, for macOS and Linux, reads over HTTP, file:// and Cloudflare R2 or any S3, with every request signed with AWS SigV4.