Error: The Replica returned an error: code 5, message: “Wasm module of canister ryjl3-tyaaa-aaaaa-aaaba-cai is not valid: Wasm module defined 6184 functions which exceeds the maximum number allowed 6000.”
We initially thought this was some sort of infinite loop, but then when we removed some of the query calls from Main.mo the error went away. I have to add them back so we’re not sure what to do …
I searched the codebase for the word “func” and got 3198 results.
The biggest motoko module, Store, has had maximum about 2000 functions in. We have 91 entity classes and we need each one to have a lot of CRUD-type functions (load, validate, loadAll etc.)
I changed the return values on create() and update() to include two function returns, and maybe that is what’s causing it. No idea why that would cause everything 2000 defined functions so break a 6k limit though.
wasm-objdump -h ./backend.wasm
backend.wasm: file format wasm 0x1
Sections:
Type start=0x0000000e end=0x00000102 (size=0x000000f4) count: 39
Import start=0x00000108 end=0x000003fb (size=0x000002f3) count: 33
Function start=0x00000401 end=0x00001c7c (size=0x0000187b) count: 6265
Table start=0x00001c82 end=0x00001c89 (size=0x00000007) count: 1
Memory start=0x00001c8f end=0x00001c92 (size=0x00000003) count: 1
Global start=0x00001c98 end=0x00001ccf (size=0x00000037) count: 10
Export start=0x00001cd5 end=0x00004e06 (size=0x00003131) count: 372
Start start=0x00004e0c end=0x00004e0e (size=0x00000002) start: 6297
Elem start=0x00004e14 end=0x0000a906 (size=0x00005af2) count: 2910
Code start=0x0000a90c end=0x001008b0 (size=0x000f5fa4) count: 6265
Data start=0x001008b6 end=0x0014a663 (size=0x00049dad) count: 3056
Custom start=0x0014a669 end=0x00208396 (size=0x000bdd2d) "name"
Custom start=0x0020839c end=0x00216377 (size=0x0000dfdb) "icp:public candid:service"
Custom start=0x0021637d end=0x00216397 (size=0x0000001a) "icp:private candid:args"
Custom start=0x0021639d end=0x002163dd (size=0x00000040) "icp:private motoko:stable-types"
Custom start=0x002163e3 end=0x00216405 (size=0x00000022) "icp:private motoko:compiler"
Custom start=0x0021640b end=0x002183be (size=0x00001fb3) "motoko"
Have you heard of microservices before?
It’s time for microcanisters!
dont really want a complete refactor of the code. Plus, this part of the software is designed to all be in one canister, just because of inter-canister call overhead.
You’re telling me that out of you thousands of functions that there’s none you can pull out (with the data of course)?
I would start by diagramming what’s directly connected and involving the frontend more in the process to coordinate calls and fetching data between your microservices. It’s a bit more work up front, but will scale much better. You’ll also find it’s easier to make small changes without breaking everything.
Microservices design tends to lend itself pretty well to a NoSQL type of storage approach with data duplication.
I’m assuming your app is just going to keep growing, so it’s only going to get harder and harder to refactor if you keep kicking the bucket down the road.
I’d recommend taking a day or so to draw out your current data model and think about how you can split out functionality.
For data transfer, you can either pull it out and backfill into the new canisters, or think about using your old canister as a database for your new canisters, pulling in data from the old canister slowly with inter-canister calls until it’s all in the new canisters (sort of like cache-miss then fetch and pull in from old to new canisters)
I can’t really pull out any functions. If anything it’s going to be over 10,000 within weeks.
We’re designing a single ORM Canister. Yes you could potentially split it between canisters but the default is that everything is on the same canister. The code is generated by Golang, which is the issue. A single entity could account for 50 or more functions.
We’re talking tiny little functions, for instance checking that a reciprocal relation is correct simply uses func(e : Entity) { e.owner };
I guess understanding why there’s a 6000 function limit is the first step.
Hi @borovan, let me try to shed some light on some of the points you raised here.
Before launching the IC, we had run some experiments to understand the impact of defining lots of globals and functions in Wasm modules. The result was that defining 10000 functions was stressing the system both in terms of additional compilation overhead (admittedly this doesn’t happen on every message execution) but also overhead on every message execution. Based on some data we had from existing canisters at the time, we decided to set a limit on number of globals and functions that could be added in a Wasm module as a safeguard.
No idea why that would cause everything 2000 defined functions so break a 6k limit though.
Just to be clear: Having 2000 functions in your Motoko/Rust canister does not guarantee that it will be compiled 1-1 to Wasm. It’s very likely that for each function in your “higher” level language there will be multiple functions created in Wasm. What we count is the number of Wasm functions (obviously, we don’t have access to your source code).
In order to help you here, we’ll need some clarifications. You mentioned something about a Main.mo which made me think that you’re developing your canister in Motoko but then later you also said
The code is generated by Golang, which is the issue.
I’m confused a bit. What code is generated by Golang and how is that integrated in your canister?
Hi, thanks for getting back to me. Here’s a quick overview of our ORM. We’re building a very complex game (WoW, Dwarf Fortress, Neopets)
DB.mo
public let rarity = NatMap.empty<E.Rarity>();
Rarity.mo
public type Rarity = ORM.Entity<R.Rarity, M.Rarity>;
Main.mo
// Rarity
public shared query({ caller }) func loadManyRarity(ids : [Nat]) : async (Result, [E.Rarity]) {
Query<R.Rarity, M.Rarity>(caller, stores.rarity, "root.admin.load").loadMany(ids);
};
public shared query({ caller }) func loadRangeRarity(start : Nat, count : Nat) : async (Result, [E.Rarity]) {
Query<R.Rarity, M.Rarity>(caller, stores.rarity, "root.admin.load").loadRange(start, count);
};
public shared({ caller }) func createRarity(r : R.Rarity) : async (Result, ?E.Rarity) {
Query<R.Rarity, M.Rarity>(caller, stores.rarity, "root").create(r);
};
public shared({ caller }) func updaSteRarity(id : Nat, r : R.Rarity) : async (Result, ?E.Rarity) {
Query<R.Rarity, M.Rarity>(caller, stores.rarity, "root").update(id, r);
};
Metadata.mo
// Rarity
public type Rarity = {
created : T.Timestamp;
modified : T.Timestamp;
};
public type RarityM = {
var created : T.Timestamp;
var modified : T.Timestamp;
};
public func mutateRarity(r : Rarity) : RarityM {
{
var created = r.created;
var modified = r.modified;
};
};
public func restoreRarity(r : RarityM) : Rarity {
{
created = r.created;
modified = r.modified;
};
};
public func newRarity(deps : Deps) : Rarity {
let value = Value.Value(deps);
{
created = value.timeNow();
modified = value.timeNow();
};
};
Record.mo
// Rarity
//
// Rarity
// We don't store weightings here because they would unbalance the game
// too much if ever changed.
//
public type Rarity = {
variant : T.Rarity;
name : Text;
description : Text;
icon : Nat;
};
public type RarityM = {
var variant : T.Rarity;
var name : Text;
var description : Text;
var icon : Nat;
};
public func mutateRarity(r : Rarity) : RarityM {
{
var variant = r.variant;
var name = r.name;
var description = r.description;
var icon = r.icon;
};
};
public func restoreRarity(r : RarityM) : Rarity {
{
variant = r.variant;
name = r.name;
description = r.description;
icon = r.icon;
};
};
public func validateRarity(r : R.Rarity) : Result {
let m = Result.Map();
m.add("name", VR.Game.name(r.name));
m.add("description", VR.Game.description(r.description));
m.result();
};
Store.mo
//
// Rarity
//
public let rarity = object {
public let name = "Rarity";
// read
public func load(id : Nat) : ?E.Rarity {
deps.db.rarity.get(id);
};
public func loadMany(ids : [Nat]) : Iter.Iter<E.Rarity> {
queryLoadMany<E.Rarity>(deps.db.rarity, ids);
};
public func loadRange(start : Nat, count : Nat) : Iter.Iter<E.Rarity> {
queryLoadRange<E.Rarity>(deps.db.rarity, start, count);
};
public func loadAll() : Iter.Iter<E.Rarity> {
deps.db.rarity.vals();
};
func validate(e : E.Rarity) : Result {
let m = Result.Map();
m.merge(R.validateRarity(e.data));
m.add("icon", hasOne<E.Icon>(deps.db.icon, e.data.icon));
m.result();
};
// create
public func create(id : Nat, r : R.Rarity) : ORM.StoreResult<E.Rarity> {
// must be null
let prev = deps.db.rarity.get(id);
if (prev != null) {
return storeResultError(#err("ID " # Nat.toText(id) # " already exists"), prev);
};
// validate
let e = {
id = id;
data = r;
metadata = M.newRarity(deps);
};
var err = validate(e);
if (err != #ok) { return storeResultError(err, null) };
// create
deps.db.rarity.put(id, e);
{
result = #ok;
entity = ?e;
onSuccess = func() {
};
onError = func() {
deps.db.rarity.delete(id);
};
};
};
// update
public func update(id : Nat, r : R.Rarity) : ORM.StoreResult<E.Rarity> {
// switch on previous value
switch (deps.db.rarity.get(id)) {
case (null) {
return storeResultError(#err("ID " # Nat.toText(id) # " not found"), null);
};
case (?prev) {
// validate
let e = {
id = id;
data = r;
metadata = updateMetadata(id);
};
var err = validate(e);
if (err != #ok) { return storeResultError(err, ?prev) };
// update
deps.db.rarity.put(id, e);
{
result = #ok;
entity = ?e;
onSuccess = func() {
};
onError = func() {
deps.db.rarity.put(id, prev);
};
};
};
};
};
// write
func updateMetadata(id : Nat) : M.Rarity {
let md = switch(deps.db.rarity.get(id)) {
case (?e) { e.metadata };
case (null) { M.newRarity(deps) };
};
let mut = M.mutateRarity(md);
mut.modified := Value.Value(deps).timeNow();
M.restoreRarity(mut);
};
};
This is exactly the system we need to build this game. The issue is that with a function limit of 6000 (after a morning’s optimisation we’re at 5998), we need a huge architectural rewrite and will probably lose a lot of the features that we’ve implemented so far.
Thanks for sharing some more of the context. Unfortunately, I don’t have a good recommendation off the top of my head for you, other than trying to optimise the code that generates the .mo files (which you already did it seems). That said, I can imagine if you’re trying to build such a complex game that such optimisations can get you only part of the way.
I can take a point on our side to look again into measuring the impact of having lots of functions defined in a Wasm module and see how much we could increase this limit without jeopardising the stability of the system. But I want to be honest with you that it might not be easy to go above 10000 (or something in that order) soon enough (according to your projection that could be a number you might need sooner than later).
It’s probably a good idea to rethink your design and whether you could split things up in a way that would alleviate this issue and be better positioned for future extensions in your dapp (it sounds like with the current design you can easily keep growing the number of functions needed which means that you’ll keep hitting the limit until (if) we can bump it again and the cycle repeats).
Hi, yeah would definitely appreciate it if you could re-run some tests and consider upping the limit. I’ll see what we can do with the code, but it is going to be a massive rewrite regardless.
Is there any reason that’s specific to Wasm that means that it can’t handle large amounts of functions? Or is this something the motoko compiler could eventually resolve? It’s really hard programming in a functional language and not using functions.
Have you looked into the ECS approach instead of writing a custom ORM?
Is there any reason that’s specific to Wasm that means that it can’t handle large amounts of functions?
I don’t think it’s something specific to Wasm per se, but I would say that it’s a combination of how we use Wasmtime on the IC and potentially other inefficiencies in Wasmtime (the runtime library we’re using to execute Wasm) itself. To put differently, if we sit down and focus our efforts on this specific aspect, we can likely find optimisations that would make the effect of having lots of functions defined less severe.
Or is this something the motoko compiler could eventually resolve?
My guess would be that it’s not completely up to the Motoko compiler either. If you have lots of functions in your .mo file to begin with, then the best you can hope is to have at most that many in the corresponding Wasm (as I hinted you typically end up with more). So, if you truly need to define lots of functions, you wouldn’t be able to just leave it up to the Motoko compiler.
I’d have to look into it more, but there’s a lot of similarity in what we’re doing. I just want a NoSQL database that has relational integrity as far as we can achieve it.
When I say ORM it’s not a traditional database with queries, it’s really just storing nested types into a TrieMap. We can do unique indexes using a BiMap, and validation of fields and relations, but that’s really it.
We’re down at 5000 after a few hours of optimising, which is nice.
I think if we can reduce the effective functions per entity to something nearer 10 then we should be ok to have an ORM Canister and move the game logic elsewhere.
Still, any increase of that value will make it a lot easier for us. I don’t think it’s going to be a constant arms race, but yes would rather avoid any horrible premature optimisation and unneeded complexity
Out of curiousity, have you tried running wasm-opt to reduce the size of the binary? It could be that a lot of these functions are never called and can be eliminated by tree shaking.
wasm-opt -O2 ./.dfx/local/canisters/backend/backend.wasm -o new.wasm
unknown name subsection at 1294587
like that? The generated wasm file is 902Kb vs. 1.7 megs. It’s removed 30% of the functions too, so we’re down to 3300 ish.
Wow. Why doesn’t this come as standard? Am I doing it right?
Why doesn’t this come as standard?
That’s a very good point. I think we should make it a standard for .wasm binaries produced by dfx. I don’t see any reason why not.
Well, it’s adding quite a lot to the trusted code base. Are we sure we want to go there?