Preupgrade and postupgrade limits

skilesare · June 14, 2021, 9:57pm

Are there cycle limits on preupgrade and postupgrade? I would imagine that large hashmaps, lists, etc being output to an array will take some significant cycles once they get pretty large. Likewise, loading a few hundred thousand items out of an array into a HashMap will take some time as well.

prithvi · June 15, 2021, 7:11pm

The cycles limits are enforced at the message level not at the pre/post upgrade canister method level. You can find the definition here:

github.com

dfinity/ic/blob/master/rs/config/src/subnet_config.rs#L11

    
      
          //! This module implements support for static configurations for components that
          //! can be different for different subnet types.
          
          
use crate::execution_environment::SUBNET_HEAP_DELTA_CAPACITY;
          use ic_base_types::NumBytes;
          use ic_registry_subnet_type::SubnetType;
          use ic_types::{Cycles, NumInstructions};
          
          
// We assume 1 cycles unit ≅ 1 CPU cycle, so on a 2 GHz CPU one message has
          // approximately 2.5 seconds to be processed.
          pub(crate) const MAX_INSTRUCTIONS_PER_MESSAGE: NumInstructions =
              NumInstructions::new((1 << 30) * 5);
          
          
/// The per subnet type configuration for the scheduler component
          #[derive(Clone)]
          pub struct SchedulerConfig {
              /// Number of canisters that the scheduler is allowed to schedule in
              /// parallel.
              pub scheduler_cores: usize,
          
          
    /// Maximum amount of instructions a single round can consume (on one

If you have inefficient serialization/deserialization of your stable memory data in those pre/post hooks, you can hit the execution message limit. If you hit this limit, your state will not be committed (it’ll roll back) but the cycles that’ve been consumed will be deducted from your canister’s cycle balance.

skilesare · June 15, 2021, 8:40pm

Thanks!

Do you know where we can find that execution limit?

prithvi · June 15, 2021, 10:26pm

You can find some of the execution limits in the file I linked above, there are subnet type specific limits defined in there.

alexeychirkov · January 5, 2022, 10:00am

Does it mean that if my canister hits a limit and preupgrade method will never succeed - I will never be able to “upgrade” my canister? The only way to change the code will be “reinstall”?

nomeata · January 5, 2022, 10:32am

Correct – let me shamelessly plug my canister audit advice about the risks related to canister upgrades.

alexeychirkov · January 5, 2022, 2:09pm

Thank for sharing a link.

What about MAX_INSTRUCTIONS_PER_INSTALL_CODE constant?

I thought that “pre/post upgrade” uses MAX_INSTRUCTIONS_PER_INSTALL_CODE but not MAX_INSTRUCTIONS_PER_MESSAGE …

Please clarify this question…

nomeata · January 6, 2022, 12:51pm

I believe you are right. The constant for that is here:

github.com

dfinity/ic/blob/65cc15f3337da9d5d712a076c0d5140d41ebc2e0/rs/config/src/subnet_config.rs#L39


      
          const MAX_INSTRUCTIONS_PER_ROUND: NumInstructions = NumInstructions::new(7 * B);
          
          // Limit per `install_code` message. It's bigger than the limit for a regular
          // update call to allow for canisters with bigger state to be upgraded.
          // This is a temporary measure until a longer term solution that alleviates the
          // limitations with the current upgrade process is implemented.
          //
          // The value is picked to allow roughly for 4GB of state to be stored to stable
          // memory during upgrade. We know that we hit `MAX_INSTRUCTIONS_PER_MESSAGE`
          // with roughly 100MB of state, so we set the limit to 40x.
          const MAX_INSTRUCTIONS_PER_INSTALL_CODE: NumInstructions = NumInstructions::new(40 * 5 * B);
          
          // The factor to bump the instruction limit for system subnets.
          const SYSTEM_SUBNET_FACTOR: u64 = 10;
          
          // The maximum amount of heap delta per iteration. Data from production shows
          // memory throughput of 100MB/s. Subnets run with different finalization rate,
          // so a round may take 1 to 4 seconds. To avoid regressing the throughput of
          // slow subnets while maintaining the speed of fast subnets, we use the middle
          // value of 200MB.
          const MAX_HEAP_DELTA_PER_ITERATION: NumBytes = NumBytes::new(200 * M);

jzxchiang · January 7, 2022, 7:42am

It’s a little disappointing to read that Motoko stable variables are not recommended for use due to:

Motoko serializing stable variables into the main wasm memory before copying the bytes into stable memory (thus making a 4 GB canister memory limit more like 2 GB)
the aforementioned instruction limit on the pre-upgrade and post-upgrade hooks

I’m really hoping these will be addressed soon, as Motoko stable variables are the first place new IC developers turn to having read the docs.

Topic		Replies	Views
Scalability of update calls in a common scenario Developers	21	3698	October 30, 2020
Heads-Up: Fixing PROD issue by adjusting the complexity of some System API calls Developers	17	1313	April 29, 2022
Canister Multithreads Roadmap	22	1388	November 15, 2021
Big Map (for scaling canisters across subnets) Roadmap Discussing	16	3854	September 29, 2021
Proposal to set canister limit Governance community-consideration	1	204	March 22, 2024

Preupgrade and postupgrade limits

Related Topics