Hi @josephgranata,
Thanks for your kind words. Feel free to ask me anything about stable structures.
Considering your questions. This is entirely possible, but there are a couple of thing you have to worry about:
1 Message size limit
IC has a limited maximum message size (last year it was something around 2MB, maybe it has changed since then, idk). This means, that when you want to upload more data (a big file), you have to split it in chunks. You may reassemble these chunks back into a single file, once they reach your canister, but it is more likely that you would want them to be split forever, since then you would have to split them again each time a user requests one of these files (the message size limit works both ways around).
This means that you’ll probably have an internal data layout like this (expressed in stable data structures):
type FileId = u64;
type ChunkId = u64;
type Chunk = Vec<u8>;
#[derive(CandidType, Deserialize, CandidAsDynSizeBytes, StableType)]
struct File {
name: String,
path: String,
chunks: Vec<ChunkId>,
}
struct State {
files: SBTreeMap<FileId, SBox<File>>,
chunks: SBTreeMap<ChunkId, SBox<Chunk>>,
}
2 Canister memory limit
When your canister only uses heap-based (standard) collections, it can occupy at most 4GB of memory. When your canister uses stable collections, it can occupy at most 40GB of memory. There are two important points here:
- “At most” doesn’t mean, that you’re guaranteed with 40GB of memory per canister. You might get unlucky and get your canister deployed to a subnet which physically has only 1GB of memory left. In that case your canister will be granted with this 1GB and that’s it.
- Even if we were guaranteed with 40GB of memory per canister, it may still be not enough, especially if we’re talking about an app that works with a lot of images/documents.
So, please, think about scalability model of your app beforehand. Think about multi-canister approaches.
In ic-stable-memory every stable collection method that may allocate additional memory returns a Result type. If the value is Ok - the allocation was successful (or there wasn’t one) and you’re good to go. But if the value is Err - this means that your canister is out of memory and you have to do something with it. For example, deploy a new canister and re-transmit all the traffic to it.
In practice it looks like that:
// using data structures from the previous code snippet
fn create_file(file_id: FileId, name: String, path: String) {
let state: State = get_state();
let file = File::new(name, path);
let boxed_file = SBox::new(file).expect("out of memory");
state.files.insert(file_id, boxed_file).expect("out of memory");
}
In this example I’m just throwing an error when we reach the memory limit, but you can run any kind of program to resolve the situation.
Just try them out, play around. Try to solve your use-case with them. There is a lot of documentation about them - you can find everything linked in the repo. Also hit me with any questions about it here or on Github.
