I believe the regular API overwrites the entire stable store as documented here.
Are there suggested approaches that I can adopt where I can store large collections of data to the stable BTreeMap and store other data selectively on the heap? I don’t want to choose between the two but would rather like to use them both in the same canister depending on the use case
Tagging @ielashi since he’s the author of the stable structures library
It is possible to use stable structures alongside storing other data from the heap. However, you cannot use the ic_cdk API since it writes to stable memory with disregard to what’s already there.
The solution here is to rely on the MemoryManager. The MemoryManager takes stable memory and provides an API that simulates having multiple memories, so you can give one memory to StableBTreeMap, and use another memory for storing data from the heap.
You can see an example of this in the Bitcoin canister, where we serialize data on the heap in memory 0, and use memory 1+ for stable structures.
This has been a common source of confusion for ic-stable-structures users. I’ll create an example of how this can be done and share it in this thread.
To clarify, candid != serde. serde is a general framework for serialization/deserialization. Libraries like candid, ciborium, or serde_json all rely on serde to support various data formats.
The candid in the example is used for the endpoints of the canister, which is the standard that you’d expect. In theory you can also use candid as the data format in pre/post upgrade, but candid doesn’t support the serde(skip) directive that we need to skip stable structures, so to use it you’ll need to implement the CandidType manually for now.
For simplicity, and to avoid having to implement CandidType manually in my example, I used CBOR as the data format in pre/post upgrade, but that choice was to some extent arbitrary.
@chenyan How difficult would be it to support serde(skip) in candid?
How difficult would be it to support serde(skip) in candid?
Not difficult. We just need to implement that attribute in the proc macro.
OTOH, it makes sense to use a non-Candid encoding scheme for upgrade, although it means yet another dependency. Candid is designed for canister communication, and may not suit perfectly for upgrade: 1) it’s not optimized for message size; 2) it doesn’t support value sharing; 3) The opt subtyping rule can be a footgun in upgrade to silently drop values of incompatible type (we have a static check in motoko, but not in Rust).