Overview
The latest 0.182 work focuses on two things:
-
Reducing SQL query overhead, especially for blob-heavy result sets.
-
Simplifying the storage/runtime matrix by removing the old direct-stable backend.
Two important changes landed:
-
SQL blob projections now stay binary until callers explicitly render them. -
The old direct-stable storage backend has been removed in favor of the journaled cached-stable path.
Blob SQL Output
Previously, SQL projection rows were eagerly converted into display strings.
That was especially expensive for blobs, because binary payloads were converted into hex text before crossing the SQL facade boundary.
That mixed up three separate concerns:
-
the executed value;
-
the API/wire representation;
-
the human-readable display format.
SQL projection results now use typed OutputValue cells instead of Vec<Vec<String>>.
Blob values remain binary through the result boundary. Hex rendering only happens when a caller explicitly asks for display/table output.
In other words:
Before: SQL value -> display string -> caller
After: SQL value -> typed OutputValue -> caller
|
└── render only when display is requested
This keeps the semantic SQL result clean while preserving human-readable rendering where it belongs: at the display boundary.
Measured Impact
Focused SQL perf matrix, current implementation vs previous baseline:
| Query shape | Before | After | Improvement |
|---|---|---|---|
| Blob payload projection | 14,135,523 | 1,202,655 | -91.49% |
| Blob thumbnail projection | 1,817,791 | 1,050,264 | -42.22% |
| Wide scalar projection | 1,138,594 | 1,122,779 | -1.39% |
| Numeric expression projection | 1,055,763 | 1,042,382 | -1.27% |
The major win comes from avoiding eager hex expansion and unnecessary display conversion work.
For one measured blob payload query:
| Output | Raw blob bytes | Old hex-rendered estimate |
|---|---|---|
| Full blob payload rows | 121,856 | 243,724 |
| Thumbnail rows | 7,168 | 14,342 |
So for blob-heavy SQL results, the new typed path avoids roughly 2x display expansion unless display output is explicitly requested.
Human-readable rendering still exists. It has simply moved out of the primary semantic result format and into the display layer.
Direct-Stable Storage Removed
IcyDB also removed the old direct-stable runtime storage lane.
This does not mean IcyDB is moving away from stable memory. Durable storage now goes through the journaled cached-stable path, which is the runtime path we want to keep optimizing and proving.
The removed path was the older direct storage(stable(...)) style backend. It increased the number of execution/storage combinations we had to test, audit, and reason about, while the journaled path is now the intended durable storage authority.
Current storage modes are now simpler:
| Mode | Purpose |
|---|---|
heap |
Volatile/test-style storage |
journaled |
Durable cached-stable storage |
Why Remove Direct Stable?
The perf audit showed that direct-stable was not pulling its weight.
Focused primary-key SQL LIMIT 1 read:
| Backend | Total instructions |
|---|---|
| Direct stable | 772,272 |
| Journaled | 663,495 |
That is a reduction of 108,777 instructions, or about 14.1%.
Cached path:
| Backend | Total instructions |
|---|---|
| Cached stable | 378,947 |
| Cached journaled | 270,860 |
That is a reduction of 108,087 instructions, or about 28.5%.
The larger matrix also showed stable-backed mirror scans as the biggest storage hotspots. In the worst rows, heap and journaled equivalents were roughly half the total instruction cost, and the store phase dropped from about 28.6M instructions to about 7.0M.
So removing direct-stable is not just a cleanup. It reduces the storage matrix, removes an underperforming branch, and lets durable execution focus on the path we actually want to harden.
Compatibility
This is a pre-1.0 hard cut.
Schemas still using direct storage(stable(...)) should move to the journaled durable storage path.
The direct-stable parser/runtime affordances are being removed rather than kept as compatibility branches.
For normal users, the intended durable setup is now journaled storage.
Blob SQL display behavior is still available, but callers consuming typed SQL results now receive blob payloads as binary values instead of pre-rendered hex strings.
What This Unlocks
This makes query performance easier to reason about because:
-
blob payload cost is now visible separately from display-rendering cost;
-
SQL result values now have a typed semantic contract;
-
the storage matrix no longer has a direct-stable branch multiplying tests and edge cases;
-
durable execution and recovery work can focus on the journaled cached-stable path.
The next major performance target is actual row/blob materialization cost: reducing how much data we decode, move, or clone when a query only needs part of a row.
That is the next layer down now that SQL output and storage-mode cleanup are no longer obscuring the measurement.
