What about taking advantage of heap memory’s performance and Motoko’s new Enhanced Orthogonal Persistence feature coupled with wasm64?
Migrating this functionality to Motoko would eliminate the need to put limitations on following since I’d anticipate that wasm64 heap memory capacity is going to grow significantly this year, and would avoid future performance bottlenecks related to traversing BTree-like data structures and repeated reads stable memory. This seems to potentially be the best way forward if performance is the primary goal.
In terms of data upgrade safety, canister snapshots help a great deal here, as copies can be taken immediately before each upgrade and restored immediately in the case of a bad upgrade.
And then in terms of the performance related to heap upgrades with a lot of data, upgrades that use Enhanced Orthogonal Persistence are instant - there’s no downtime related to serialization to stable memory or deserialization back from stable memory into heap.
This would also be a great opportunity to start building more core system canisters in Motoko, the language of the Internet Computer, as well as to help the language mature and drive it toward corporate adoption/and have more examples to generate AI code from.
I’d be happy to help consult and contribute to this initiative if there’s interest from the foundation.