Heartbeat improvements / Timers [Community Consideration]

I can see you are using 0.6.2 version, change it to 0.7.5 in vessel.dhall file.

@h1teshtr1path1
Error: Failed to download Motoko binaries for version 0.7.5, with “403 Forbidden”

Details: <?xml version="1.0" encoding="UTF-8"?>
AccessDeniedAccess DeniedVBR4JNP82AWKXBW6SGSGkPB1HNWrn68r8vsVaUlKID562wMHUeiIaUF59OhcuJASiTED8j7wqtfGfTCAE7kjGTidt3s=
I get this

this is my packageset as well

and the other file
image

Should be fixed in the just released ic-cdk v0.6.9, i.e. periodic timers can now cancel themselves.

2 Likes

Sorry, I’m not an expert in Motoko, maybe @ggreif could provide more details here?

But I really suggest you to wait for the official dfx release. The release is being a bit delayed due to a critical issue. But hopefully it will be resolved soon and released. I’ll keep you updated.

@rvanasa you just happened to release for 0.7.5 and 0.7.6, should folks just retry?

Yep! The official vessel package-set is now updated to 0.7.6.

As a quick debugging step for anyone running into issues, try removing everything in your package-set.dhall file and then pasting the following link:

https://github.com/dfinity/vessel-package-set/releases/download/mo-0.7.6-20230120/package-set.dhall

This will configure your project to use the latest official package-set without any overrides or modifications.

As the new maintainer for vessel, I will do my best to make this process a lot simpler in the near future. Please feel free to submit an issue on GitHub if you run into any problems, since we are actively collecting feedback for a possible top-down redesign of the package manager with a focus on improving UX and scalability.

3 Likes

Looks like I cannot use this inside a timer job function if the timer is set inside an actor class body. Is there a reason for this? The compiler tells me “cannot use this before this has been defined” but it seems like this should already be defined at the point when a timer job is run.

Giving us a glimpse at the code would help. I have seen such an error and the solution was to move a function around. Maybe this helps :-\

If I place it inside postupgrade hook, it works. But I would need something like postinstall instead.

@claudio and I are discussing this. Can you separate out the third argument to the call to setTimer to become a private actor method?

(Claudio started FR: treating actor self references as initially defined · Issue #3718 · dfinity/motoko · GitHub for this issue.)

1 Like

It doesn’t work either.

Then (in the meantime) I suggest to have an actor-level var to hold a ?Principal and initialise that at some convenient place. Then use it from your timed action.

Quick update. There is a dfx version that contains moc v0.7.6:

DFX_VERSION=0.13.0-beta.2 sh -ci "$(curl -fsSL https://internetcomputer.org/install.sh)"

Have fun!

3 Likes

Heads-up on timers: Each timer call takes a slot in your canister’s message queue, and they will fail when you reach the limit.

[Canister ryjl3-tyaaa-aaaaa-aaaba-cai] in canister_global_timer: SysTransient: Maximum queue capacity 500 reached

Found this out while trying to max out the output queues with Notify calls. It looks like you’d need to have a “watchdog” timer if you want to dynamically add timers based on received calls. Designing around this seems tricky.

Question for Dfinity folks: is there a way for a canister to check the current size of the queue?

Timers library tries to call, and the call fails. Not sure how knowing the queue size would help here…

Instead, we can think of improving the call error handling. At the moment we just log the error: cdk-rs/timer.rs at main · dfinity/cdk-rs · GitHub

What do you think would be a better approach?

Oh, I think I see what you mean. I was calling notify from a closure passed to the timer so this would have been the err caught on the line you linked. Let me try passing a fn that actually checks if the notify succeeds and re-adds it as a timer if it fails. I’ll report back.

Thanks for the prompt answer!

1 Like

Something weird is happening with timers when reaching the “Maximum queue capacity 500 reached” error. I’ve uploaded a minimum repro case here: GitHub - GLicDEV/test_timers

Calling testTimers(1) produces the expected results:

[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] Iteration: 0 - Added a timer with timer_id: TimerId(4v3)
[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] Iteration: 0 - Got called from timer!

The same for testTimers(499) - although this might need a dfx canister deposit-cycles 10000000000000 test_timers_backend to cover the cycles needed.

[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] Iteration: 0 - Added a timer with timer_id: TimerId(8v5)
[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] Iteration: 1 - Added a timer with timer_id: TimerId(2v5)
[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] Iteration: 2 - Added a timer with timer_id: TimerId(6v5)

*snip*

[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] Iteration: 497 - Added a timer with timer_id: TimerId(7v3)
[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] Iteration: 498 - Added a timer with timer_id: TimerId(3v3)
[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] Iteration: 0 - Got called from timer!
[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] Iteration: 2 - Got called from timer!

*snip*

[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] Iteration: 1 - Got called from timer!
[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] Iteration: 3 - Got called from timer!

(ordering not relevant, but all timers get called from 0 to 498)

However calling testTimers(500)+ does this:

(+dfx canister deposit-cycles 10000000000000 test_timers_backend)

[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] Iteration: 0 - Added a timer with timer_id: TimerId(1v1)

*snip*

[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] Iteration: 499 - Added a timer with timer_id: TimerId(500v1)
[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] in canister_global_timer: SysTransient: Maximum queue capacity 500 reached
[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] in canister_global_timer: SysTransient: Maximum queue capacity 500 reached
[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] in canister_global_timer: SysTransient: Maximum queue capacity 500 reached
[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] in canister_global_timer: SysTransient: Maximum queue capacity 500 reached
[Canister rrkah-fqaaa-aaaaa-aaaaq-cai] in canister_global_timer: SysTransient: Maximum queue capacity 500 reached

None of the 500 timers get to the call_from_timer fn.

Changing the line where I add timers to ic_cdk::timer::set_timer(Duration::new(i as u64, 0), move || call_from_timer(i)); works, as 1-2 timers will be called each round over the next 500 seconds.

Looking through the code it seems that on this line there’s a While{}. Do you think it would be prudent to add some bounds to that so that it doesn’t attempt to process more tasks than it can? Or break at the first error and re-attempt them in a later round? One would expect that at least some timers should be executed, even if not all of them can be.

1 Like

I think there are bugs, thanks for reporting.

The fix is on its way: fix: RUN-527: retry timers execution by dfinity-berestovskyy · Pull Request #366 · dfinity/cdk-rs · GitHub

2 Likes