StableBTreeMap Key Deserialization Fails with UTF-8 Panic

Developers Rust
,
1

Hey everyone :waving_hand:

I’m currently investigating a deserialization error that occurs when using a StableBTreeMap. The serialization/deserialization is handled using the candid crate, and the issue arises after a larger number of entries (100+ rounds) are stored.

Here’s some context on how I store data:

pub struct TokenSymbol(String); // There are only 3 tokens used

const MAX_VALUE_SIZE: u32 = 12; // Max string length used - 6 chars, so this should be enough in theory

impl Storable for TokenSymbol {
    fn to_bytes(&self) -> Cow<[u8]> {
        let mut buf = vec![];
        minicbor::encode(self, &mut buf).expect("event encoding should always succeed");
        Cow::Owned(buf)
    }

    fn from_bytes(bytes: Cow<[u8]>) -> Self {
        minicbor::decode(bytes.as_ref())
            .unwrap_or_else(|e| panic!("failed to decode event bytes {}: {e}", hex::encode(bytes)))
    }

    const BOUND: Bound = Bound::Bounded {
        max_size: MAX_VALUE_SIZE,
        is_fixed_size: false,
    };
}


#[derive(Serialize, Deserialize)]
pub struct PaymentProcessor {
    #[serde(skip, default = "init_map")]
    round_history: StableBTreeMap<(TokenSymbol, u16), PaymentRound, VM>, // u16 is a payment round id
}

So in tests after having more that 100 rounds stored (with random tokens combination) I got such error:

Panicked at 'called Result::unwrap() on an Err value: Custom(Fail to decode argument 0 from text to text

Caused by:
    invalid utf-8 sequence of 1 bytes from index 5)', backend/libraries/types/src/token.rs:44:31
2031-05-06 07:00:00.000022091 UTC: [Canister lqy7q-dh777-77777-aaaaq-cai] in canister_global_timer: CanisterError: IC0503: Error from Canister lqy7q-dh777-77777-aaaaq-cai: Canister called ic0.trap with message: Panicked at 'called Result::unwrap() on an Err value: Custom(Fail to decode argument 0 from text to text

Caused by:
    invalid utf-8 sequence of 1 bytes from index 5)', backend/libraries/types/src/token.rs:44:31.
Consider gracefully handling failures from this canister or altering the canister to handle exceptions.

Questions

  • What might be causing this error? Is it a bug or could it be due to incorrect or unsafe use of the candid crate?
  • Is there any known limitation on key size or encoding format for StableBTreeMap?

Thanks in advance!

1 Like
2

Hi Victoria,
Do you think you’d be able to push somewhere a minimal reproducible example?

1 Like
3

I see minicbor, not candid used for encoding/decoding in above example.

With minicbor, I’d expect to see some attributes in your code, which seems to be missing. Example minicbor struct for a single property (transparent): internet-identity/src/internet_identity/src/storage/storable/account_reference_list.rs at main · dfinity/internet-identity · GitHub There are various other minicbor examples in this storable directory.

As for encoding strings, you shouldn’t need either minicbor or candid, instead you can directly use my_string.into_bytes() and String::from_utf8(my_bytes) assuming you only want to store the string without the wrapping struct.

2 Likes
4

Sure, you can take a look at this repo:

You can use this script to add pocket_ic bin
scripts/init-integration-testing-env.sh
And then one to build the canister and run the integration test:
scripts/run_tests.sh
After running this integration test for some time - errors are starting to appear

There is also a unit test, which also fails but with another error:

1 Like
5

Yes, you are right, I accidentally pasted the wrong snippet, here is what I used initially:

impl Storable for TokenSymbol {
    fn to_bytes(&self) -> Cow<[u8]> {
        Cow::Owned(Encode!(self).unwrap())
    }
    fn from_bytes(bytes: Cow<[u8]>) -> Self {
        Decode!(&bytes, Self).unwrap()
    }
    const BOUND: Bound = Bound::Bounded {
        max_size: MAX_VALUE_SIZE,
        is_fixed_size: false,
    };
}

We will anyway proceed with another solution and use enum, something like this:

#[derive(
    Debug, Serialize, Clone, Deserialize, CandidType, PartialEq, Eq, Hash, PartialOrd, Ord,
)]
#[repr(u8)]
enum TokenSymbol {
    ICP = 0,
}

But for the future I still would like to know what might caused those errors

6

I wasn’t able to get the integration test running correctly, but the unit test is failing with the assertion error tuples.rs:24:17: assertion failed: a_bytes.len() <= a_max_size . This clearly indicates that the token symbol GLDGov exceeds the 12-byte limit. Increasing MAX_VALUE_SIZE should resolve this issue.

Note that Candid-encoded symbol GLDGov is: DIDL\0\u{1}q\u{6}GLDGov