Stable Structures: Removing the Bounded Size Requirement

Developers
21

Nice! I’ll have a play around with it shortly.

If we have maps where the keys and values are fixed size would they also benefit from using v2 or should they remain on v1?

22

As far as beta testing is concerned, you won’t observe much of a difference. When V2 is ready and we make a production release, developers will not actually choose between V2 and V1 - that’ll be abstracted away. BTreeMaps would all automatically get upgraded to V2 under the hood.

3 Likes
23

Which of the main options discussed in the OP were selected for the beta? I don’t see that the example is showing how to define any of the traits

24

We opted for solution #2, which was the favored solution in this thread. All the examples have been updated with the new API already. Here’s an example on how to implement Storable for a type: https://github.com/dfinity/stable-structures/blob/main/examples/src/custom_types_example/src/lib.rs

1 Like
25

I see it’s all on main. This is extremely exciting! I’m hoping to get a chance to dive in soon, actually I was just in the middle of reimplementing stable structures in Azle.

1 Like
26

So far so good, got our first set of tests passing with the unbounded types

1 Like
27

Islam any news on support for this feature from the Motoko team at DFINITY? @claudio is this on the Motoko team radar? Any estimates when it would make it to the code base?

It is great news, great job fellows, I am just wishing it works in Motoko too.

28

Hey Joseph, I’m unaware of this specific change being ported to Motoko. AFAIK there is a Motoko port of StableBTreeMap that’s been done by the community and is not owned by the Motoko team. I do know that the Motoko team are working on a memory manager to allow giving separate virtual memories to a type, which is a necessary building block for supporting something like stable structures there.

1 Like
29

We’ve just release Motoko 0.10.0 which contains a new library Region.mo for declaring isolated subregions of IC Stable Memory, similar to the isolated Memories of stable structures but with a little more integration into the type system of Motoko.

We don’t have anything equivalent StableBTree yet, but @matthewhammer is investigating adapting the work of @sardariuss (a Motoko port of the original Rust StableBTree, AFAIU) to use Region.mo for better encapsulation.

3 Likes
30

Good news @ielashi and @claudio so we don’t have the Rust equivalent functionality yet, but there is work happening, and I am glad for StableBTreeMap it has been a lifesaver, and it makes the whole upgrade process far easier.

@claudio I will be looking forward for Motoko 0.10 and I am hopeful Regions enable a close equivalent to Rust’s Stable Memory features, including one day the Bounded Size updates.

31

Dragginz is now using this, was an easy integration. Will let you know if we have any problems, thanks again!

5 Likes
32

When trying to implement this with version 0.6.0-beta.0 i ran into the issue where;

  • the WHITELIST data isn’t persistent between canister upgrades
  • the WHITELIST_ID is persistent between canister upgrades

Anything known what could cause this / what I am doing wrong?

type Memory = VirtualMemory<DefaultMemoryImpl>;
type RMemory = RestrictedMemory<DefaultMemoryImpl>;

thread_local! {
    static MEMORY_MANAGER: RefCell<MemoryManager<DefaultMemoryImpl>> =
        RefCell::new(MemoryManager::init(DefaultMemoryImpl::default()));

    pub static WHITELIST_ID: RefCell<StableCell<u64, RMemory>> = RefCell::new(
        StableCell::init(
            RMemory::new(DefaultMemoryImpl::default(), 0..MAX_PAGES),
            0,
        ).expect("failed")
    );
    pub static WHITELISTS: RefCell<StableBTreeMap<u64, Whitelist, Memory>> = RefCell::new(
        StableBTreeMap::init_v2(
            MEMORY_MANAGER.with(|m| m.borrow().get(MemoryId::new(0))),
        )
    );
}

Whitelist struct

#[derive(CandidType, Deserialize, Clone, Debug)]
pub struct Whitelist {
    pub id: u64,
    pub name: String,
    pub color: String,
    pub owner: Principal,
    pub whitelist: Vec<Principal>,
}

impl Storable for Whitelist {
    fn to_bytes(&self) -> std::borrow::Cow<[u8]> {
        Cow::Owned(Encode!(self).unwrap())
    }

    fn from_bytes(bytes: std::borrow::Cow<[u8]>) -> Self {
        Decode!(bytes.as_ref(), Self).unwrap()
    }

    const BOUND: Bound = Bound::Unbounded;
}
33

The problem lies in how you’re structuring stable memory.

  • You’re first initializing MEMORY_MANAGER, giving it the entire stable memory.
  • You’re initializing WHITELIST_ID, also giving it the entire stable memory (a restricted memory from 0..MAX_PAGES is practically all the stable memory that there is).

This clash is why you’re seeing only WHITELIST_ID being persisted, while other structures are not persisted. I recommend that you keep it simple and use the memory manager for everything.

Another approach would be to give WHITELIST_ID a RMemory with the range 0..1 and the MEMORY_MANAGER a RMemory with the range 1..MAX_PAGES.

And, do you need to store the WHITELIST_ID separately? Isn’t that information already stored in WHITELISTS?

1 Like
34

Thanks, That makes sense, i assumed it was a memory allocation for that specific storage.

The reason why i keep a seperate WHITELIST_ID is because i assume it is cheaper to increment this id compared to searching for the highest value of the whitelist.id

35

You can use the last_key_value method on the BTreeMap. That’s an efficient call.

1 Like