Heartbeat improvements / Timers [Community Consideration]

Should be fixed in the just released ic-cdk v0.6.9, i.e. periodic timers can now cancel themselves.

2 Likes

Sorry, I’m not an expert in Motoko, maybe @ggreif could provide more details here?

But I really suggest you to wait for the official dfx release. The release is being a bit delayed due to a critical issue. But hopefully it will be resolved soon and released. I’ll keep you updated.

@rvanasa you just happened to release for 0.7.5 and 0.7.6, should folks just retry?

Yep! The official vessel package-set is now updated to 0.7.6.

As a quick debugging step for anyone running into issues, try removing everything in your package-set.dhall file and then pasting the following link:

https://github.com/dfinity/vessel-package-set/releases/download/mo-0.7.6-20230120/package-set.dhall

This will configure your project to use the latest official package-set without any overrides or modifications.

As the new maintainer for vessel, I will do my best to make this process a lot simpler in the near future. Please feel free to submit an issue on GitHub if you run into any problems, since we are actively collecting feedback for a possible top-down redesign of the package manager with a focus on improving UX and scalability.

4 Likes

Looks like I cannot use this inside a timer job function if the timer is set inside an actor class body. Is there a reason for this? The compiler tells me “cannot use this before this has been defined” but it seems like this should already be defined at the point when a timer job is run.

Giving us a glimpse at the code would help. I have seen such an error and the solution was to move a function around. Maybe this helps :-\

If I place it inside postupgrade hook, it works. But I would need something like postinstall instead.

@claudio and I are discussing this. Can you separate out the third argument to the call to setTimer to become a private actor method?

(Claudio started FR: treating actor self references as initially defined · Issue #3718 · dfinity/motoko · GitHub for this issue.)

1 Like

It doesn’t work either.

Then (in the meantime) I suggest to have an actor-level var to hold a ?Principal and initialise that at some convenient place. Then use it from your timed action.

Quick update. There is a dfx version that contains moc v0.7.6:

DFX_VERSION=0.13.0 sh -ci "$(curl -fsSL https://internetcomputer.org/install.sh)"

Have fun!

4 Likes

Heads-up on timers: Each timer call takes a slot in your canister’s message queue, and they will fail when you reach the limit.

[Canister ryjl3-tyaaa-aaaaa-aaaba-cai] in canister_global_timer: SysTransient: Maximum queue capacity 500 reached

Found this out while trying to max out the output queues with Notify calls. It looks like you’d need to have a “watchdog” timer if you want to dynamically add timers based on received calls. Designing around this seems tricky.

Question for Dfinity folks: is there a way for a canister to check the current size of the queue?

Timers library tries to call, and the call fails. Not sure how knowing the queue size would help here…

Instead, we can think of improving the call error handling. At the moment we just log the error: cdk-rs/timer.rs at main · dfinity/cdk-rs · GitHub

What do you think would be a better approach?

Oh, I think I see what you mean. I was calling notify from a closure passed to the timer so this would have been the err caught on the line you linked. Let me try passing a fn that actually checks if the notify succeeds and re-adds it as a timer if it fails. I’ll report back.

Thanks for the prompt answer!

1 Like

Something weird is happening with timers when reaching the “Maximum queue capacity 500 reached” error. I’ve uploaded a minimum repro case here: GitHub - GLicDEV/test_timers

Calling testTimers(1) produces the expected results:

[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] Iteration: 0 - Added a timer with timer_id: TimerId(4v3)
[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] Iteration: 0 - Got called from timer!

The same for testTimers(499) - although this might need a dfx canister deposit-cycles 10000000000000 test_timers_backend to cover the cycles needed.

[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] Iteration: 0 - Added a timer with timer_id: TimerId(8v5)
[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] Iteration: 1 - Added a timer with timer_id: TimerId(2v5)
[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] Iteration: 2 - Added a timer with timer_id: TimerId(6v5)

*snip*

[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] Iteration: 497 - Added a timer with timer_id: TimerId(7v3)
[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] Iteration: 498 - Added a timer with timer_id: TimerId(3v3)
[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] Iteration: 0 - Got called from timer!
[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] Iteration: 2 - Got called from timer!

*snip*

[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] Iteration: 1 - Got called from timer!
[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] Iteration: 3 - Got called from timer!

(ordering not relevant, but all timers get called from 0 to 498)

However calling testTimers(500)+ does this:

(+dfx canister deposit-cycles 10000000000000 test_timers_backend)

[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] Iteration: 0 - Added a timer with timer_id: TimerId(1v1)

*snip*

[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] Iteration: 499 - Added a timer with timer_id: TimerId(500v1)
[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] in canister_global_timer: SysTransient: Maximum queue capacity 500 reached
[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] in canister_global_timer: SysTransient: Maximum queue capacity 500 reached
[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] in canister_global_timer: SysTransient: Maximum queue capacity 500 reached
[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] in canister_global_timer: SysTransient: Maximum queue capacity 500 reached
[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] in canister_global_timer: SysTransient: Maximum queue capacity 500 reached

None of the 500 timers get to the call_from_timer fn.

Changing the line where I add timers to ic_cdk::timer::set_timer(Duration::new(i as u64, 0), move || call_from_timer(i)); works, as 1-2 timers will be called each round over the next 500 seconds.

Looking through the code it seems that on this line there’s a While{}. Do you think it would be prudent to add some bounds to that so that it doesn’t attempt to process more tasks than it can? Or break at the first error and re-attempt them in a later round? One would expect that at least some timers should be executed, even if not all of them can be.

1 Like

I think there are bugs, thanks for reporting.

The fix is on its way: fix: RUN-527: retry timers execution by dfinity-berestovskyy · Pull Request #366 · dfinity/cdk-rs · GitHub

2 Likes

hey folks,
We have a new Rust example comparing timers vs heartbeats: examples/rust/periodic_tasks at master · dfinity/examples · GitHub

Please let me know If you see any typos/bugs or if something needs to be explained better.

4 Likes

The timers documentation is complete:

  1. Developer Guide: Backend: Periodic Tasks and Timers
    The language agnostic description of timers and heartbeats. Probably, the most important section is Timers Library Limitations, please check it out.

  2. For Motoko: Motoko Developer Guide: Timers

  3. For Rust: Developer Guide: Backend: Rust: Using Timers tutorial

Please let me know if we’re missing something.

9 Likes

Quick question - Does setting or cancelling a timer have a cycles cost? I’m aware that the timer calling it’s task does.

There is no fee associated with timer setting/cancelling. It’s rather is a normal Canister execution (there are libraries for Motoko and Rust), so each library call takes some (rather tiny) amount of Instructions.

The precise number of Instructions depend on the language, library version, compilation options etc. There is a performance counter to measure precisely the number of Instructions consumed by a Canister code snippet.

Note, there is an update message execution fee on IC, which is 590K cycles, and timer execution is an update message. But setting/cancelling the timer is just a few Instructions, there is no fee for that…

Here is the list of the IC fees for the reference: Internet Computer Loading

3 Likes