The 4GB memory limit on canisters is one of the major complications I’m running into while building on top of the Internet Computer, especially while building fundamental infrastructural tools.
Orthogonal persistence is a beautiful dream, but the memory limit and the necessity of splitting data structures and apps across multiple canisters for scaling…somewhat hinders that dream.
I’m wondering if it would be possible for this limit to be abstracted away below the canister level, so that it is not exposed to developers. Considering that the limit has to be dealt with somewhere, it seems an explicit choice was made to let the developers deal with it. If we can deal with it as developers, then couldn’t the platform also deal with it, but perhaps in a more generalized manner?
If this is at all possible, I believe it would truly simplify so many things for developers for years to come.
I of course don’t know much about the internals of the system, but couldn’t something like virtual memory on our operating systems work for canisters? Instead of canisters directly accessing Wasm module memory, perhaps a virtual memory system could be created that automatically scales up and down according to the needs of canisters, and canisters could interact directly with that virtual memory system.
It would just be such a dream to create something like a HashMap, and just keep pumping values into it forever without any thought to 4GB limits and cross-canister calls and managing all of that craziness.