Efficient data structure for a large data

Hi dfinity team,
I am dealing with a larg data which is currently in hashMap with key pair, in query call i have to filter over it to filter specific items and some other operation like sort it by date and then pagination, with passege of the time data is more than 1 million record and my query call trape with error that more than 5 billion instructions exceed something like this. Do you have any solution for it and also tell me that which data structure should be i used instead of hashmap

What is the exact error that you are having? What is the programming language that you are using (I’m assuming Rust)?

i am using motoko,
error was like this
Replica Error: reject code CanisterError, reject message Error from Canister bkyz2-fmaaa-aaaaa-qaaaq-cai: Canister exceeded the instruction limit for single message execution

Got it. Yes, this error can show if you’re trying to insert a large volume of data into your canister, or in your case, your data is arranged in an inefficient manner.

What’s the exact data set? Is there a way that you can more efficiently get the data without so many loops?

Some teams have used a TrieMap instead but I wouldn’t resort to just using it. You should structure your data in a way where it doesn’t go to a bunch of a loops.

stableheapbtreemap is a BTree (sorted map) that is efficient for key sorted data, as all operations are O(logn). We’ve been using it in production with CycleOps for almost two years, and for building secondary search indexes on several other ICP apps as well.

FYI filtering isn’t ever efficient unless you create indices for your data.

So if you have a large enough dataset, you’ll need to plan out your data model to support efficient indexes queries.

1 Like