# Error storing in ic stable structure

**URL:** <https://forum.dfinity.org/t/error-storing-in-ic-stable-structure/40197>\
**Category:** Developers\
**Created:** [January 17, 2025, 2:07pm UTC](https://forum.dfinity.org/t/error-storing-in-ic-stable-structure/40197 "2025-01-17T14:07:19Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![AliSci](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/alisci/32/33799_2.png) [@AliSci](https://forum.dfinity.org/u/AliSci)\
**Post date:** [January 17, 2025, 2:07pm UTC](https://forum.dfinity.org/t/error-storing-in-ic-stable-structure/40197/1 "2025-01-17T14:07:19Z")

</div>

first of all, I tried to save short content data to my backend and it was fine. But when the content get bigger i can’t save anymore. Also, Could that happend because I migrate some field before like adding new field or removing field in the struct ? I think that because locally i ran `dfx deploy backend --mode=reinstall` and the error gone. But on the IC i don’t wanna lose my data.

```tsx
[Canister bkyz2-fmaaa-aaaaa-qaaaq-cai] Panicked at 'Attempting to allocate an already allocated chunk.', /Users/ahmed/.cargo/registry/src/index.crates.io-6f17d22bba15001f/ic-stable-structures-0.6.5/src/btreemap/allocator.rs:166:9

```

 ![Screenshot 2025-01-17 at 10.02.58 PM](https://us1.discourse-cdn.com/flex023/uploads/dfn/original/3X/2/4/244c26cad973e11307e8f85e6561334c228cae56.png)  
my code

```rs
#[derive(Clone, Debug, Deserialize, CandidType)]
pub struct ContentNode {
    pub id: ContentId,
    pub parent: Option<ContentId>,
    pub _type: String,
    pub value: String,
    pub text: String,
    pub language: String,
    pub indent: u64,
    pub data: Option<ContentData>,
    pub listStyleType: String,
    pub listStart: u64,
    #[serde(default)]
    pub children: Vec<ContentId>,
}

pub type ContentTree = Vec<ContentNode>;
#[derive(Clone, Debug, Deserialize, CandidType)]
pub struct ContentNodeVec {
    pub contents: HashMap<FileId, ContentTree>,
}

static FILE_CONTENTS: RefCell<StableBTreeMap<String, ContentNodeVec, Memory>> = RefCell::new(
        StableBTreeMap::init(
            MEMORY_MANAGER.with(|m| m.borrow().get(MemoryId::new(5))),
        )
    );

pub fn update_file_contents(file_id: FileId, content_nodes: ContentTree) {
        FILE_CONTENTS.with(|file_contents| {
            let mut contents = file_contents.borrow_mut();
            let mut content_map = HashMap::new();
            content_map.insert(file_id.clone(), content_nodes);
            contents.insert(
                file_id,
                ContentNodeVec {
                    contents: content_map,
                },
            );
        });
    }

```

```rs

impl Storable for ContentNodeVec {
    fn to_bytes(&self) -> Cow<[u8]> {
        Cow::Owned(Encode!(self).unwrap())
    }

    fn from_bytes(bytes: Cow<[u8]>) -> Self {
        Decode!(bytes.as_ref(), Self).unwrap_or_else(|_| ContentNodeVec {
            contents: HashMap::new(),
        })
    }

    const BOUND: Bound = Bound::Unbounded;
}

```

---

<div class="post-metadata">

**Author:** ![kristofer](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/kristofer/32/11135_2.png) [@kristofer](https://forum.dfinity.org/u/kristofer)\
**Post date:** [January 18, 2025, 10:40pm UTC](https://forum.dfinity.org/t/error-storing-in-ic-stable-structure/40197/2 "2025-01-18T22:40:18Z")

</div>

I believe the `ContentNodeVec` needs to be a bounded type. Storing a `HashMap` in your `StableBTreeMap` is probably not a good idea.

If you haven’t read it I highly recommend this blog post:

> **[Tutorial: stable-structures](https://mmapped.blog/posts/14-stable-structures#stable-btree)**
>
> An introduction to the stable-structures library.

> The [`BTreeMap`](https://docs.rs/ic-stable-structures/0.4.0/ic_stable_structures/btreemap/struct.BTreeMap.html) stable structure is an associative container that can hold any [bounded storable type](https://mmapped.blog/posts/14-stable-structures#storable-types). The map must know the sizes of the keys and values because it allocates nodes from a pool of fixed-size tree nodes.

Have you considered compiling your canister as WASI, with the support of [wasi2ic](https://github.com/wasm-forge/wasi2ic)? Then you could store your files using standard file system functions.

---

<div class="post-metadata">

**Author:** ![AliSci](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/alisci/32/33799_2.png) [@AliSci](https://forum.dfinity.org/u/AliSci)\
**Post date:** [January 19, 2025, 2:26pm UTC](https://forum.dfinity.org/t/error-storing-in-ic-stable-structure/40197/3 "2025-01-19T14:26:51Z")

</div>

> [@kristofer](#):
>
> bounded

Hi, thank you for helping again.

do u mean

```rs
use ic_stable_structures::{BoundedStorable,Storable};

```

`impl BoundedStorable for ContentNodeVec {`?

---

<div class="post-metadata">

**Author:** ![berestovskyy](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/berestovskyy/32/3989_2.png) [@berestovskyy](https://forum.dfinity.org/u/berestovskyy)\
**Post date:** [January 19, 2025, 3:04pm UTC](https://forum.dfinity.org/t/error-storing-in-ic-stable-structure/40197/4 "2025-01-19T15:04:13Z")

</div>

```auto
Panicked at 'Attempting to allocate an already allocated chunk.',

```

From [the source stable-structures code](https://github.com/dfinity/stable-structures/blob/69ed47f9b5001af67d650c714cd56ec3ee0ef2bb/src/btreemap/allocator.rs#L166):

```auto
        // Get the next available chunk.
        let chunk_addr = self.free_list_head;
        let mut chunk = ChunkHeader::load(chunk_addr, &self.memory);

        // The available chunk must not be allocated.
        assert!(
            !chunk.allocated,
            "Attempting to allocate an already allocated chunk."
        );

```

This indicates that a data chunk is present in the free list while still being marked as allocated. This suggests a corruption in the allocator’s internal state. Data recovery in such a situation may be challenging…

---

<div class="post-metadata">

**Author:** ![kristofer](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/kristofer/32/11135_2.png) [@kristofer](https://forum.dfinity.org/u/kristofer)\
**Post date:** [January 19, 2025, 6:56pm UTC](https://forum.dfinity.org/t/error-storing-in-ic-stable-structure/40197/5 "2025-01-19T18:56:49Z")

</div>

No, I meant, it is generally not a good idea to store an item with unknown size (`ContentNodeVec`) as the value of a `StableBTreeMap`. Correct me if I am wrong @berestovskyy.

---

<div class="post-metadata">

**Author:** ![AliSci](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/alisci/32/33799_2.png) [@AliSci](https://forum.dfinity.org/u/AliSci)\
**Post date:** [January 21, 2025, 7:53am UTC](https://forum.dfinity.org/t/error-storing-in-ic-stable-structure/40197/6 "2025-01-21T07:53:36Z")

</div>

I just added ` const BOUND: Bound = Bound::Bounded { max_size: 9999999, is_fixed_size: false, };` but still not getting any results

---

<div class="post-metadata">

**Author:** ![kristofer](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/kristofer/32/11135_2.png) [@kristofer](https://forum.dfinity.org/u/kristofer)\
**Post date:** [January 21, 2025, 8:31am UTC](https://forum.dfinity.org/t/error-storing-in-ic-stable-structure/40197/7 "2025-01-21T08:31:50Z")

</div>

I think you should consider rethinking the storage strategy and mimic that of a regular file system.

Separate the metadata structure from the actual blocks of a file.

### Metadata structure

- Stored as a stable `BTreeMap<filename, metadata>`
- Contains file name etc
- Contains a block index, a fixed size vector with pointers to the file blocks

This works well if you have a fixed maximum file size that is not huge. If number of file blocks exceed what you have allocated for the metadata structure you would need to be able to link to another metadata structure. More advanced block indexes could be used as well, b-trees etc.

### File blocks

- Stored as a stable `Vec<file_block>` where each block is 4KB or something
- On save, a file is divided up into blocks and saved, metadata structure updated with the pointers (indexes in the file block vector)

---

<div class="post-metadata">

**Author:** ![sea-snake](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/sea-snake/32/7243_2.png) [@sea-snake](https://forum.dfinity.org/u/sea-snake)\
**Post date:** [January 21, 2025, 8:36am UTC](https://forum.dfinity.org/t/error-storing-in-ic-stable-structure/40197/8 "2025-01-21T08:36:37Z")

</div>

> [@kristofer](#):
>
> not a good idea to store an item with unknown size (`ContentNodeVec`)

If I remember correctly, that limitation was lifted. It now stores values that are too big like a linked list under the hood as far as I’ve read.

Still curious to know what caused the possible stable structure data corruption. @AliSci if you manage to make this reproducible, please post the steps here.

For now, only way forward I could think of would be to migrate the stored values to another memory in a scheduled timer moving X values at a time.

---

<div class="post-metadata">

**Author:** ![kristofer](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/kristofer/32/11135_2.png) [@kristofer](https://forum.dfinity.org/u/kristofer)\
**Post date:** [January 21, 2025, 8:59am UTC](https://forum.dfinity.org/t/error-storing-in-ic-stable-structure/40197/9 "2025-01-21T08:59:59Z")

</div>

> [@sea-snake](#):
>
> If I remember correctly, that limitation was lifted. It now stores values that are too big like a linked list under the hood as far as I’ve read.

Any idea where you read that?

Docs and library both references the tutorial by Roman that states:

> [@kristofer](#):
>
> The [`BTreeMap`](https://docs.rs/ic-stable-structures/0.4.0/ic_stable_structures/btreemap/struct.BTreeMap.html) stable structure is an associative container that can hold any [bounded storable type](https://mmapped.blog/posts/14-stable-structures#storable-types). The map must know the sizes of the keys and values because it allocates nodes from a pool of fixed-size tree nodes.

---

<div class="post-metadata">

**Author:** ![AliSci](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/alisci/32/33799_2.png) [@AliSci](https://forum.dfinity.org/u/AliSci)\
**Post date:** [January 21, 2025, 9:10am UTC](https://forum.dfinity.org/t/error-storing-in-ic-stable-structure/40197/10 "2025-01-21T09:10:35Z")

</div>

@kristofer can you give me any code example please? do u mean  
making small blocks of the actual data . So the issue is a matter of size only

```rs

    static FILE_CONTENTS: RefCell<StableBTreeMap<String, ContentNodeVec, Memory>> = RefCell::new(
        StableBTreeMap::init(
            MEMORY_MANAGER.with(|m| m.borrow().get(MemoryId::new(5))),
        )
    );

#[derive(Clone, Debug, Deserialize, CandidType)]
struct FileBlock {
    index: u32,
    data: Vec<u8>,
}

```

---

<div class="post-metadata">

**Author:** ![sea-snake](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/sea-snake/32/7243_2.png) [@sea-snake](https://forum.dfinity.org/u/sea-snake)\
**Post date:** [January 21, 2025, 9:17am UTC](https://forum.dfinity.org/t/error-storing-in-ic-stable-structure/40197/11 "2025-01-21T09:17:32Z")

</div>

> [@kristofer](#):
>
> Any idea where you read that?

> [@Stable Structures: Removing the Bounded Size Requirement](https://forum.dfinity.org/t/stable-structures-removing-the-bounded-size-requirement/21167):
>
> Hey everyone, [Stable structures](https://github.com/dfinity/stable-structures) is a Rust library built by DFINITY that allows canister developers to store tens of GiBs of data without the need for upgrade hooks. We’ve received [feedback](https://github.com/dfinity/stable-structures/issues/69) from users that having to put an upper bound on the size of whatever can be stored in a StableBTreeMap makes the library difficult to use, and in many cases is a blocker for adoption altogether. This limitation is also shared by other stable structures, like StableVec. Currently, to store a type in a Stable…

---

<div class="post-metadata">

**Author:** ![ielashi](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/ielashi/32/15704_2.png) [@ielashi](https://forum.dfinity.org/u/ielashi)\
**Post date:** [January 21, 2025, 9:36am UTC](https://forum.dfinity.org/t/error-storing-in-ic-stable-structure/40197/12 "2025-01-21T09:36:11Z")

</div>

> [@AliSci](#):
>
> `[Canister bkyz2-fmaaa-aaaaa-qaaaq-cai] Panicked at 'Attempting to allocate an already allocated chunk.', /Users/ahmed/.cargo/registry/src/index.crates.io-6f17d22bba15001f/ic-stable-structures-0.6.5/src/btreemap/allocator.rs:166:9`

This error indicates that there’s some kind of significant corruption to the StableBTreeMap. Did you ever change the bounds of your key/value structs? Unfortunately you cannot change the bounds of structs once you start storing them in a StableBTreeMap, and doing so could corrupt your data.

---

<div class="post-metadata">

**Author:** ![AliSci](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/alisci/32/33799_2.png) [@AliSci](https://forum.dfinity.org/u/AliSci)\
**Post date:** [January 21, 2025, 9:45am UTC](https://forum.dfinity.org/t/error-storing-in-ic-stable-structure/40197/13 "2025-01-21T09:45:32Z")

</div>

> [@AliSci](#):
>
> `ContentNodeVec`

I did changed the bound but not of the content vector. However, it still work fine, but content vector does not work anymore.

 ![Screenshot 2025-01-21 at 5.41.54 PM](https://us1.discourse-cdn.com/flex023/uploads/dfn/original/3X/a/e/ae1b4f86885724d63cc14a859847612cf24f7023.png)

---

<div class="post-metadata">

**Author:** ![AliSci](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/alisci/32/33799_2.png) [@AliSci](https://forum.dfinity.org/u/AliSci)\
**Post date:** [January 21, 2025, 9:47am UTC](https://forum.dfinity.org/t/error-storing-in-ic-stable-structure/40197/14 "2025-01-21T09:47:15Z")

</div>

# How to make sure this never happen in future?

## How to get better error message instead of that undefined things ?

Ok how to reset only one of my statics, I want to chose this and empty it.

 ![Screenshot 2025-01-21 at 5.46.41 PM](https://us1.discourse-cdn.com/flex023/uploads/dfn/original/3X/c/1/c1a5b3b0381afa041047fea7e2af23eaa3adb4e8.png)

---

<div class="post-metadata">

**Author:** ![kristofer](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/kristofer/32/11135_2.png) [@kristofer](https://forum.dfinity.org/u/kristofer)\
**Post date:** [January 21, 2025, 9:50am UTC](https://forum.dfinity.org/t/error-storing-in-ic-stable-structure/40197/15 "2025-01-21T09:50:48Z")

</div>

@ielashi if we put the upgrade issues aside, is the storage layout as described by @AliSci ok?

Looks like `FILE_CONTENTS` can be simplified to store a `Vec<ContentNode>` instead of a `ContentNodeVec` with a `HashMap` containing the `Vec<ContentNode>`.

ps. We need to update docs to better describe the behaviour of `Unbounded`.

---

<div class="post-metadata">

**Author:** ![AliSci](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/alisci/32/33799_2.png) [@AliSci](https://forum.dfinity.org/u/AliSci)\
**Post date:** [January 21, 2025, 9:58am UTC](https://forum.dfinity.org/t/error-storing-in-ic-stable-structure/40197/16 "2025-01-21T09:58:36Z")

</div>

> [@kristofer](#):
>
> Vec

I can’t do `RefCell<StableBTreeMap<String, Vec<Anything>, Memory>>` because storable is not implement for Vec

---

<div class="post-metadata">

**Author:** ![ielashi](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/ielashi/32/15704_2.png) [@ielashi](https://forum.dfinity.org/u/ielashi)\
**Post date:** [January 21, 2025, 10:00am UTC](https://forum.dfinity.org/t/error-storing-in-ic-stable-structure/40197/17 "2025-01-21T10:00:11Z")

</div>

> [@AliSci](#):
>
> I did changed the bound but not of the content vector. However, it still work fine, but content vector does not work anymore.
> 
> ![Screenshot 2025-01-21 at 5.41.54 PM](https://us1.discourse-cdn.com/flex023/uploads/dfn/original/3X/a/e/ae1b4f86885724d63cc14a859847612cf24f7023.png)

Moving from bounded to unbounded is fine and is supported, but what I meant is changing the `max_size` - that would cause corruption in your data. If all you did was change the size to `Unbounded` then this, in theory, shouldn’t cause any issues.

> [@kristofer](#):
>
> if we put the upgrade issues aside, is the storage layout as described by @AliSci ok?

I’d echo your statement that the schema can be made simpler and more efficient. It’s not exactly clear to me what the key of `FILE_CONTENTS` is. Is that the file name? Also, are there many file IDs for a single file?

---

<div class="post-metadata">

**Author:** ![kristofer](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/kristofer/32/11135_2.png) [@kristofer](https://forum.dfinity.org/u/kristofer)\
**Post date:** [January 21, 2025, 10:54am UTC](https://forum.dfinity.org/t/error-storing-in-ic-stable-structure/40197/18 "2025-01-21T10:54:02Z")

</div>

Yeah, sorry.

```rust
pub struct File {
    pub contents: Vec<ContentNode>
}

impl Storable for File {
  //...
}

```

This have nothing to do with your issues it seems though.

---

<div class="post-metadata">

**Author:** ![AliSci](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/alisci/32/33799_2.png) [@AliSci](https://forum.dfinity.org/u/AliSci)\
**Post date:** [January 21, 2025, 2:58pm UTC](https://forum.dfinity.org/t/error-storing-in-ic-stable-structure/40197/19 "2025-01-21T14:58:51Z")

</div>

the key is the user id  
for each user there is a list of file\_contents, I don’t think that matter now.

---

<div class="post-metadata">

**Author:** ![AliSci](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/alisci/32/33799_2.png) [@AliSci](https://forum.dfinity.org/u/AliSci)\
**Post date:** [January 21, 2025, 3:17pm UTC](https://forum.dfinity.org/t/error-storing-in-ic-stable-structure/40197/20 "2025-01-21T15:17:58Z")

</div>

> [@AliSci](#):
>
> first of all, I tried to save short content data to my backend and it was fine.

I updated my quastion `first of all, I tried to save short content data to my backend and it was fine.`  
Does `const BOUND: Bound = Bound::Unbounded; ` mean i can store unlimited data

[Next page](https://forum.dfinity.org/t/error-storing-in-ic-stable-structure/40197.md?page=2)