decode gzip frames, and ask the server for them #107

Merged

yuris merged 1 commit from feat-gzip into main

2026-08-22 05:06:20 +00:00

yuris commented

2026-08-22 05:00:32 +00:00

Owner

The server compresses any message over 1KiB and its default scheme is brotli, which is why connect has always pinned compression=None — an omitted parameter is a wire no Defold build can read. Gzip it can read, so it now asks for that and inflates.

The fact this turned on

zlib.inflate learned to sniff the 1f 8b wrapper and strip it in Defold 1.12.4. I checked across releases rather than trust the commit subject, which reads as a routine miniz bump:

release	gzip path in `engine/dlib/src/dlib/zlib.cpp`
1.11.0, 1.12.0, 1.12.3	absent
1.12.4, 1.13.0	present (`IsGZip`, `0x8b`)

That is now the documented floor. We pin 1.13.0 in flake.nix, so CI would never have noticed the difference.

Shape

connect asks for Gzip when a gunzip is in reach, None when it is not — so a session that cannot inflate still gets a readable wire instead of a broken one. Never omitted.
decode_server(data, inflate) takes the gunzip as an argument rather than reaching for the zlib global, keeping the module's no engine APIs promise. init.lua resolves the ambient one the same way the builder resolves json.
Brotli now raises by name rather than hiding behind a generic "compressed frame" message. It is a wire this client cannot read, not a feature nobody got round to.

The tag byte replaces the message's leading byte instead of prefixing the stream, so the gzip magic sits at byte 2 and the plaintext is the ServerMessage from its sum tag.

Proof

A fixture pair — the same 711-row readback captured off a live 2.8.2 twice, plain (33,338 B) and gzipped (14,526 B) — asserted to decode to the same tag, same table, and byte-identical rows. Both plausible slice errors were mutation-proven to fail it:

mutation	result
inflate the whole frame, tag included	`truncated read of 1 byte(s) at offset 0`
drop a byte after inflating	`array count 67108864 exceeds 33323 remaining`

The first conformance run failed, which is the useful part. Every frame in phases 1–5 and 7 is under the threshold, so six phases went green before the bulk subscription arrived compressed and hit gzip frame but no inflate function was passed — the smoke's own timing wrapper had dropped the second argument. That is exactly the shape of bug this feature invites, and only a live run over 1KiB finds it.

What it buys

payload	plain	gzip
33 KiB subscription readback	33,338 B	14,526 B
479 KiB subscription readback	478,921 B	203,745 B

Below 1KiB nothing changes, so ordinary per-transaction updates are untouched — this earns its keep on the initial subscription and nowhere else. Threshold confirmed by sweeping it: 941 B arrives plain, 1171 B arrives gzipped, matching COMPRESS_THRESHOLD = 1024.

On the v1 JSON wire the parameter changes nothing at all (measured identical across None/Gzip/Brotli), so v1 passes no gunzip.

Verification

Suite green, stylua clean, and all four conformance suites pass against a live server — native v2 and v1, HTML5 v2 and v1. The HTML5 legs are what prove the engine's real zlib.inflate works in wasm, which was the one thing I could not settle by reading.

Version goes to 3.1.0: additive, since nobody is on an older engine to break.

The server compresses any message over 1KiB and its **default scheme is brotli**, which is why `connect` has always pinned `compression=None` — an omitted parameter is a wire no Defold build can read. Gzip it *can* read, so it now asks for that and inflates. ## The fact this turned on `zlib.inflate` learned to sniff the `1f 8b` wrapper and strip it in **Defold 1.12.4**. I checked across releases rather than trust the commit subject, which reads as a routine miniz bump: | release | gzip path in `engine/dlib/src/dlib/zlib.cpp` | |---|---| | 1.11.0, 1.12.0, 1.12.3 | absent | | 1.12.4, 1.13.0 | present (`IsGZip`, `0x8b`) | That is now the documented floor. We pin 1.13.0 in `flake.nix`, so CI would never have noticed the difference. ## Shape - `connect` asks for `Gzip` when a gunzip is in reach, `None` when it is not — so a session that cannot inflate still gets a readable wire instead of a broken one. Never omitted. - `decode_server(data, inflate)` takes the gunzip as an argument rather than reaching for the `zlib` global, keeping the module's *no engine APIs* promise. `init.lua` resolves the ambient one the same way the builder resolves `json`. - Brotli now raises **by name** rather than hiding behind a generic "compressed frame" message. It is a wire this client cannot read, not a feature nobody got round to. The tag byte *replaces* the message's leading byte instead of prefixing the stream, so the gzip magic sits at byte 2 and the plaintext is the `ServerMessage` from its sum tag. ## Proof A fixture pair — the same 711-row readback captured off a live 2.8.2 twice, plain (33,338 B) and gzipped (14,526 B) — asserted to decode to the same tag, same table, and byte-identical rows. Both plausible slice errors were mutation-proven to fail it: | mutation | result | |---|---| | inflate the whole frame, tag included | `truncated read of 1 byte(s) at offset 0` | | drop a byte after inflating | `array count 67108864 exceeds 33323 remaining` | **The first conformance run failed, which is the useful part.** Every frame in phases 1–5 and 7 is under the threshold, so six phases went green before the bulk subscription arrived compressed and hit `gzip frame but no inflate function was passed` — the smoke's own timing wrapper had dropped the second argument. That is exactly the shape of bug this feature invites, and only a live run over 1KiB finds it. ## What it buys | payload | plain | gzip | |---|---|---| | 33 KiB subscription readback | 33,338 B | 14,526 B | | 479 KiB subscription readback | 478,921 B | 203,745 B | Below 1KiB nothing changes, so ordinary per-transaction updates are untouched — this earns its keep on the initial subscription and nowhere else. Threshold confirmed by sweeping it: 941 B arrives plain, 1171 B arrives gzipped, matching `COMPRESS_THRESHOLD = 1024`. On the **v1 JSON wire the parameter changes nothing at all** (measured identical across None/Gzip/Brotli), so v1 passes no gunzip. ## Verification Suite green, stylua clean, and all four conformance suites pass against a live server — native v2 and v1, HTML5 v2 and v1. The HTML5 legs are what prove the engine's real `zlib.inflate` works in wasm, which was the one thing I could not settle by reading. Version goes to **3.1.0**: additive, since nobody is on an older engine to break.

yuris added 1 commit

2026-08-22 05:00:32 +00:00

decode gzip frames, and ask the server for them

ci / ci (pull_request) Successful in 21s

Details

conformance / conformance (v1) (pull_request) Successful in 2m23s

Details

conformance / conformance (v2) (pull_request) Successful in 2m20s

Details

107cdaae89

The server compresses any message over 1KiB and its DEFAULT scheme is brotli
(Compression::default, client-api-messages/src/websocket/common.rs), which is
why connect has always pinned compression=None: an omitted parameter is a wire
no Defold build can read. Gzip it can read, and has been able to since Defold
1.12.4, where zlib.inflate learned to sniff the 1f8b wrapper and strip it
(engine/dlib/src/dlib/zlib.cpp). Checked across releases rather than trusting
the commit subject, which reads as a miniz bump: 1.12.3 has no gzip path,
1.12.4 does. That is now the floor, and it is in the README.

connect asks for Gzip when a gunzip is in reach and None when it is not, so a
session that cannot inflate still gets a readable wire rather than a broken one.
decode_server takes the gunzip as an argument rather than reaching for the
zlib global, keeping the module's no-engine-APIs promise; init.lua resolves the
ambient one exactly as the builder resolves json. Brotli now raises by name
instead of hiding behind a generic 'compressed frame' message, because it is a
wire this client cannot read rather than a feature not got round to.

The tag byte REPLACES the message's leading byte instead of prefixing the
stream, so the gzip magic sits at byte 2 and the plaintext is the ServerMessage
starting at its sum tag. Pinned by a fixture pair: the same 711-row readback
captured off a live 2.8.2 twice, once plain (33338 B) and once gzipped
(14526 B), asserted to decode to the same tag, the same table, and byte-identical
rows. Both plausible slice errors were mutation-proven to fail it - inflating
the whole frame including the tag, and dropping a byte after inflating.

Cost of getting it wrong is invisible until it matters: every frame in phases 1
through 5 and 7 is under the threshold, so the first run failed in phase 6 with
'gzip frame but no inflate function was passed' - the smoke's own timing wrapper
had dropped the second argument. Six phases of green before the bulk
subscription arrived compressed. The wrapper forwards it now, and says why.

Bandwidth, measured against 2.8.2 on the default wire: a 33KiB subscription
readback arrives in 14KiB, a 479KiB one in 204KiB. Below 1KiB nothing changes,
so ordinary per-transaction updates are untouched - this earns its keep on the
initial subscription and nowhere else. On the v1 JSON wire the parameter
changes nothing at all (measured identical across None/Gzip/Brotli), so v1
passes no gunzip.

Verified: suite green, stylua clean, and all four conformance suites - native v2
and v1, HTML5 v2 and v1 - pass against a live server, which is what exercises
the engine's real zlib.inflate including in wasm.

yuris merged commit 0df1628356 into main

2026-08-22 05:06:20 +00:00

yuris referenced this pull request from a commit

2026-08-22 05:06:22 +00:00

decode gzip frames, and ask the server for them (#107)

yuris referenced this pull request

2026-08-22 07:09:57 +00:00

stdb: drop the v1 JSON wire, and ship it as 4.0.0 #110