StableBTreeMap Key Deserialization Fails with UTF-8 Panic

Developers Rust
,
1

Hey everyone :waving_hand:

I’m currently investigating a deserialization error that occurs when using a StableBTreeMap. The serialization/deserialization is handled using the candid crate, and the issue arises after a larger number of entries (100+ rounds) are stored.

Here’s some context on how I store data:

pub struct TokenSymbol(String); // There are only 3 tokens used

const MAX_VALUE_SIZE: u32 = 12; // Max string length used - 6 chars, so this should be enough in theory

impl Storable for TokenSymbol {
    fn to_bytes(&self) -> Cow<[u8]> {
        let mut buf = vec![];
        minicbor::encode(self, &mut buf).expect("event encoding should always succeed");
        Cow::Owned(buf)
    }

    fn from_bytes(bytes: Cow<[u8]>) -> Self {
        minicbor::decode(bytes.as_ref())
            .unwrap_or_else(|e| panic!("failed to decode event bytes {}: {e}", hex::encode(bytes)))
    }

    const BOUND: Bound = Bound::Bounded {
        max_size: MAX_VALUE_SIZE,
        is_fixed_size: false,
    };
}


#[derive(Serialize, Deserialize)]
pub struct PaymentProcessor {
    #[serde(skip, default = "init_map")]
    round_history: StableBTreeMap<(TokenSymbol, u16), PaymentRound, VM>, // u16 is a payment round id
}

So in tests after having more that 100 rounds stored (with random tokens combination) I got such error:

Panicked at 'called Result::unwrap() on an Err value: Custom(Fail to decode argument 0 from text to text

Caused by:
    invalid utf-8 sequence of 1 bytes from index 5)', backend/libraries/types/src/token.rs:44:31
2031-05-06 07:00:00.000022091 UTC: [Canister lqy7q-dh777-77777-aaaaq-cai] in canister_global_timer: CanisterError: IC0503: Error from Canister lqy7q-dh777-77777-aaaaq-cai: Canister called ic0.trap with message: Panicked at 'called Result::unwrap() on an Err value: Custom(Fail to decode argument 0 from text to text

Caused by:
    invalid utf-8 sequence of 1 bytes from index 5)', backend/libraries/types/src/token.rs:44:31.
Consider gracefully handling failures from this canister or altering the canister to handle exceptions.

Questions

  • What might be causing this error? Is it a bug or could it be due to incorrect or unsafe use of the candid crate?
  • Is there any known limitation on key size or encoding format for StableBTreeMap?

Thanks in advance!

1 Like
2

Hi Victoria,
Do you think you’d be able to push somewhere a minimal reproducible example?

3

I see minicbor, not candid used for encoding/decoding in above example.

With minicbor, I’d expect to see some attributes in your code, which seems to be missing. Example minicbor struct for a single property (transparent): internet-identity/src/internet_identity/src/storage/storable/account_reference_list.rs at main · dfinity/internet-identity · GitHub There are various other minicbor examples in this storable directory.

As for encoding strings, you shouldn’t need either minicbor or candid, instead you can directly use my_string.into_bytes() and String::from_utf8(my_bytes) assuming you only want to store the string without the wrapping struct.

1 Like