The documentation now has two more sections:
Hope you’ll find them useful.
Hello!
I’m trying to use a SHashMap with [u8, 136] as a key however I’m hitting all sorts of problems. The main issue now seems to be with Candid serialization/ deserialization… which is a bit beyond my rust skills.
Has anyone got a working example of SHashMap with a [u8, N] type key?
Code below for context:
#[derive( Deserialize, Serialize, CandidType, StableType, Debug, Hash, Eq, PartialEq)]
struct IDKey([u8; 136]);
impl AsFixedSizeBytes for IDKey {
const SIZE: usize = 136;
type Buf = [u8; Self::SIZE]; // use Vec<u8> for generics
fn as_fixed_size_bytes(&self, buf: &mut [u8]) {
let key_bytes = self.0.as_slice();
buf[0] = key_bytes.len() as u8;
buf[1..(1 + key_bytes.len())].copy_from_slice(key_bytes);
}
fn from_fixed_size_bytes(buf: &[u8]) -> Self {
let key_len = buf[0] as usize;
let key: &[u8] = &buf[1..(1 + key_len)];
return IDKey(key.try_into().unwrap());
}
}
#[derive(AsFixedSizeBytes, Deserialize, StableType, Debug)]
pub struct Directory {
pub id_to_ref: SHashMap<IDKey, u32>,
pub ref_to_id: SHashMap<u32, IDKey>,
pub next_ref: u32,
}
I know I could use an SBox but I’m trying to avoid this as the Directory will have a lot of lookups/ writes in my code and speed is key.
For info folks - Managed to get it to compile… I didn’t realise [u8; Size] was considered a generic as the type was known. I swapped [u8, Size] for Vec and the errors went away.
#[derive(CandidType, Deserialize, StableType, Hash, Eq, PartialEq, Clone)]
pub struct IDKey(pub Vec<u8>);
impl AsFixedSizeBytes for IDKey {
const SIZE: usize = 135;
type Buf = Vec<u8>; // use for generics
fn as_fixed_size_bytes(&self, buf: &mut [u8]) {
let key_bytes = self.0.as_slice();
buf[0] = key_bytes.len() as u8;
buf[1..(1 + key_bytes.len())].copy_from_slice(key_bytes);
}
fn from_fixed_size_bytes(buf: &[u8]) -> Self {
let key_len = buf[0] as usize;
let key: &[u8] = &buf[1..(1 + key_len)];
return IDKey(key.try_into().unwrap());
}
}
#[derive(StableType, AsFixedSizeBytes)]
pub struct Directory {
pub id_to_ref: SHashMap<IDKey, u32>,
pub ref_to_id: SHashMap<u32, IDKey>,
pub next_ref: u32,
}
Yea, AsFixedSizeBytes is implemented for any [u8; N] by default. So you could simply use the derive macro:
#[derive(AsFixedSizeBytes, Deserialize, Serialize, CandidType, StableType, Debug, Hash, Eq, PartialEq)]
struct IDKey([u8; 136]);
And keep it with an array, instead of vec.
But it seems like serde does not support const generics yet, so in order to implement CandidType/Deserialize you would have to do something like this:
#[derive(AsFixedSizeBytes, StableType, Debug, Hash, Eq, PartialEq)]
struct IDKey([u8; 136]);
impl CandidType for IDKey {
fn _ty() -> candid::types::Type {
Vec::<u8>::_ty()
}
fn idl_serialize<S>(&self, serializer: S) -> Result<(), S::Error>
where
S: candid::types::Serializer,
{
self.0.idl_serialize(serializer)
}
}
impl<'de> Deserialize<'de> for IDKey {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: serde::Deserializer<'de>,
{
let arr = Vec::<u8>::deserialize(deserializer)?.try_into().unwrap();
Ok(IDKey(arr))
}
}
The serialized is fine, but the deserialized will lose some of the performance due to an allocation.
But this is better in terms of performance, than to use Vec<u8> as your key, since most of the time it will work exclusively on stack and only allocate when deserialized (when received as an argument to a canister function).
Thats awesome! Thank you 
Another day another question 
I’ve got the stable memory working well however I’m interested in how the stable memory works along with runtime state and if it’s possible to persist runtime state over upgrades without overwriting or damaging memory allocated for stable structures.
Code for reference:
#[derive(StableType, AsFixedSizeBytes, Debug, Default)]
pub struct Main {
pub canister_data: CanisterSettings,
pub processed_data: u64,
pub directory_data: Directory,
}
#[derive(CandidType, Default, Clone)]
pub struct RuntimeState{
pub latest_txs: BlockHolder,
pub canister_logs: Vec<LogEntry>,
pub temp_vec_ptx: Vec<ProcessedTX>,
pub temp_vec_stx: Vec<SmallTX>
}
thread_local! {
pub static RUNTIME_STATE: RefCell<RuntimeState> = RefCell::default();
pub static STABLE_STATE: RefCell<Option<Main>> = RefCell::default();
}
pub fn state_init(){
stable_memory_init();
// init stable state
let mut stable_data = Main::default();
let default_admin = string_to_idkey(&"2vxsx-fae".to_string()).unwrap();
let default_canister_name = string_to_idkey(&"Name Me Please!".to_string()).unwrap();
stable_data.canister_data.authorised.push(default_admin).expect("Out of memory");
stable_data.canister_data.canister_name = default_canister_name;
STABLE_STATE.with(|state| {
*state.borrow_mut() = Some(stable_data);
});
// init runtime state
let mut runtime_state = RuntimeState::default();
runtime_state.latest_txs.init();
RUNTIME_STATE.with(|state| {
*state.borrow_mut() = runtime_state;
});
log("Canister Initialised");
}
pub fn state_pre_upgrade(){
let state: Main = STABLE_STATE.with(|s| s.borrow_mut().take().unwrap());
let boxed_state = SBox::new(state).expect("Out of memory");
store_custom_data(0, boxed_state);
stable_memory_pre_upgrade().expect("Out of memory");
}
pub fn state_post_upgrade(){
stable_memory_post_upgrade();
let state: Main = retrieve_custom_data::<Main>(0).unwrap().into_inner();
STABLE_STATE.with(|s| {
*s.borrow_mut() = Some(state);
});
}
The RUNTIME_STATE structs have a lot of Strings/ Vecs and other dynamically sized stuff which I’d like to keep on the heap rather than in stable memory. Is this possible?
Thanks in advance!
Nathan.
Hey Nathan,
You can store your RUNTIME_STATE the same way you’re storing your STABLE_STATE.
Just put it inside an SBox and use store_custom_data() function with another id.
Since you don’t implement StableType and AsDynSizeBytes for RuntimeState, I assume, you can’t do that for some reason. So, in this situation, the way you can resolve it is to simply encode/decode it manually, using candid encoding functions: encode_one() and decode_one(). Then you can put the resulting byte array into the SBox.
// I didn't check the following code, before writing it here
// if it doesn't work, please let me know
pub fn state_pre_upgrade(){
let state: Main = STABLE_STATE.with(|s| s.borrow_mut().take().unwrap());
let boxed_state = SBox::new(state).expect("Out of memory");
store_custom_data(0, boxed_state);
// RefCell supports .take() just like Option does
let rstate = RUNTIME_STATE.take();
let bytes = encode_one(rstate).expect("Unable to candid encode");
let boxed_bytes = SBox::new(bytes).expect("Out of memory");
store_custom_data(1, boxed_bytes);
stable_memory_pre_upgrade().expect("Out of memory");
}
pub fn state_post_upgrade(){
stable_memory_post_upgrade();
let state: Main = retrieve_custom_data::<Main>(0).unwrap().into_inner();
STABLE_STATE.with(|s| {
*s.borrow_mut() = Some(state);
});
let bytes: Vec<u8> = retrieve_custom_data(1).unwrap().into_inner();
let rstate: RuntimeState = decode_one(&bytes).expect("Unable to candid decode");
RUNTIME_STATE.replace(rstate);
}
Hey Buddy - Thanks for your help once again!
I’ve had a go with this but still hitting a bit of an issue. Got compile errors for take() and replace() being unstable but got around those by using the following code
pub fn state_pre_upgrade(){
// Stable Storage
let state: Main = STABLE_STATE.with(|s| s.borrow_mut().take().unwrap());
let boxed_state = SBox::new(state).expect("Out of memory");
store_custom_data(0, boxed_state);
// Runtime Storage
let rstate = RUNTIME_STATE.with(|s|{s.borrow_mut().to_owned()});
let bytes = encode_one(rstate).expect("Unable to candid encode");
let boxed_bytes = SBox::new(bytes).expect("Out of memory");
store_custom_data(1, boxed_bytes);
stable_memory_pre_upgrade().expect("Out of memory");
}
pub fn state_post_upgrade(){
stable_memory_post_upgrade();
let state: Main = retrieve_custom_data::<Main>(0).unwrap().into_inner();
STABLE_STATE.with(|s| {
*s.borrow_mut() = Some(state);
});
// Runtime Storage
let bytes: Vec<u8> = retrieve_custom_data::<RuntimeState>(1).unwrap().into_inner();
let rstate: RuntimeState = decode_one(&bytes).expect("Unable to candid decode");
RUNTIME_STATE.with(|s| {
*s.borrow_mut() = rstate;
});
}
However this then throws an error for .into_inner() for trait bounds not satisfied - AsDynSizeBytes, and StableType.
So I had a bash at deriving both for RuntimeState. AsDynSizeBytes wasn’t a problem as I can use SBoxes to wrap the dynamic stuff, however I hit a bit of a dead end with StableType.
RuntimeState has a VecDeque which I don’t think is supported at all. Removing this from the struct still gives issues with any Vec which contains a struct. I think that StableType only compiles if the type is rust standard type (string etc) or principal type?
It’s not a massive issue if I can’t save RUNTIME_STATE during upgrades in this canister… but would be a nice-to-have.
Interested in your thoughts,
Cheers,
Nathan.
The problem is in this line
let bytes: Vec<u8> = retrieve_custom_data::<RuntimeState>(1).unwrap().into_inner();
Change it like this
let bytes: Vec<u8> = retrieve_custom_data(1).unwrap().into_inner();
You are no longer storing RuntimeState in the stable memory, you are now storing Vec<u8>.
Slight change… but yes it works 
let bytes: Vec<u8> = retrieve_custom_data::<Vec<u8>>(1).unwrap().into_inner();
Thank you!