Deterministic Time Slicing

[icme]

@ulan

Yes, we are currently at 2x limit. DTS looks good so far in production, so I think we could go to 6x relatively quickly: in a couple of replica versions.

This is awesome @ulan thanks for the update - can’t wait to test it out!

@ulan

Once the canister reaches 4GB and updates start failing due to out-of-memory, then upgrades will also fail. This means that the canister becomes stuck without any fix.

Just to be clear, this memory limit is because of overflowing the heap/main memory, and is different than the upgrades failing due to upgrade cycles limitations, correct? I believe streaming serialization was implemented (see @claudio’s comment in link) that allows very close to (slightly less than) 4GB of heap memory to be serialized to stable memory during upgrades.

So, is the actual storage we can use in one canister 2GB at most due to upgrades?

The 2GB limit is due to the default, semi-space, copying garbage collector, which roughly requires twice the heap space to perform GC, but conveniently leaves the second half available for stable variable serialization. In moc 0.6.27, we’ve implemented streaming serialization of stable variables to stable memory, avoiding the need for all but a smallish buffer in wasm memory. The hope is that streaming would allow people to more safely select the alternative mark-compact gc, which can utilize …

Also, with respect to the “freezing threshold” idea

@ulan

I have an idea to solve this problem by introducing a “freezing threshold” for memory. It would be a canister settings parameter with the default value of 3GB. When the canister reaches that limit, then updates start failing, but upgrades continue to work. The owner of the canister would be able to increase or decrease the parameter.

I think it would be great if this would be a system func “hook” that a canister developer could tie into and trigger an action once this threshold is hit.

With CanDB I’m doing something similar (but not implemented at the language/system level obviously). I currently have two fixed limits that are lower than the 4GB heap limits. These limits are:

1. An INSERT_THRESHOLD, after which no new items can be inserted
2. An UPDATE_THRESHOLD, after which no new items can be modified (i.e prevents a user from appending to the attribute metadata associated with a record)

I use these thresholds to both trigger auto-scaling actions as well as to permit or reject additional inserts/updates to the CanDB data store in a canister.