Meet CycleOps. Proactive, automated, no-code canister management for the Internet Computer

[icme]

Hello IC Developer Community

Today we’re very proud to announce the public release of CycleOps!

CycleOps is the very first proactive, automated, no-code canister management tool on the IC, empowering devs with cycles balance metrics while allowing them to keep full control of their applications .

Ready for stress free cycles management? Get started at https://cycleops.dev today!

Welcome to CycleOps

Core Features

• Proactive, periodic monitoring - CycleOps runs a “cron”, using the canister timers feature of the Internet Computer to periodically monitor canisters.
• Automated topups & email notifications - When a canister runs low on cycles, CycleOps proactively tops up your canister for you and sends you a summary straight to your email inbox .
• Accounting & downloadable transaction history - Taxes on your mind? CycleOps maintains a complete history of all customer transactions, and provides you with a downloadable CSV to make things easier for both you and your accountant
• Historical canister cycles charts - View time series of your canisters’ historical cycles balance data and analyze usage trends

• No-code integration - get set up without adding a single line of code to your canisters
• Competitive cycles market rate - To top it all off, CycleOps purchases cycles in bulk from @sonic_ooo or the cheapest swap marketplace, meaning that developers can save both time and money by switching to Geico…I mean CycleOps

Why do I need Cycles Management?

The IC’s reverse gas model is great for users, but this means that applications and developers are responsible for funding their canisters with Cycles and keeping them “topped up”.

Developers can set a cycles balance freezing threshold that acts as a defense against a cycle drain attack , such that once a canister’s cycles balance hits the freezing threshold the canister will become responsive. Once a canister is “frozen” , it will reject subsequent requests while continuing to burn cycles based on it’s own memory utilization https://twitter.com/ICDevs_org/status/1617614801508646912?s=20

Unfortunately, if a canister runs completely out of cycles, it is deleted from the IC - including the wasm and all associated state . This is what happened with the IC Turtles project in 2022

How does CycleOps Work?

As opposed to hosting an application on AWS or through a centralized cloud provider, CycleOps is an on-chain monitoring solution that utilizes a black-holed , immutable canister smart contract to monitor 3rd party canisters. This blackhole is only capable of monitoring other canisters through the management canister’s canister_status API, but don’t just take our word for it ! The source code for this “Balance Checker” canister is publicly available, which you can find at GitHub - CycleOperators/BalanceCheckerVerification: Blackhole and wasm verification of the CycleOps StatusChecker canister. The repository has instructions for how you can verify the CycleOps blackhole locally or through an easy-to-use GitHub action .

Is CycleOps reliable?

• The CycleOps alpha has been humming along smoothly for the last 2 months without any downtime or user reported bugs
• Before launching the CycleOps alpha, CycleOps received a code review from @nomeata that allowed us to evaluate and harden the security of our codebase
• CycleOps is a multi-canister application that is sustained, by CycleOps. For redundancy, our team has built a full monitoring suite that automates regular health checks and alerts of CycleOps itself.

What’s next for CycleOps?

The CycleOps project has received a follow-up grant from DFINITY , and has many features in the pipeline, including:

Features in progress:

• Canister cycles usage spike alerts , notifying teams when anomalous cycles burn occurs
• An open source Cycles management library

Planned features (feedback requested):

• Teams support (multiple principals per account, assign roles & access control)
• Bring your own cycles (deposit cycles, not just ICP!)

We have other ideas in the pipeline, but want to hear from you! What canister management features, integration, and support does your application need?

Additional Questions? Check out our resources!

• Getting Started with CycleOps
• Best Practices for Top-up Rules
• Integrating Canisters with CycleOps
• FAQ

P.S.

The CycleOps team, (myself @icme, Jorgen @jorgenbuilder, and Quint @quint) wholeheartedly thank our alpha testers for their invaluable feedback , and have kicked off an ICP airdrop as appreciation to all of the alpha participants that made it through the onboarding process!

[Bugslayer]

Good Product, All the best Cycleops Team

[icme]

Application traffic on the internet is inherently spiky, but sometimes if left unchecked this can rack up costs or signal a greater issue for a canister, such as an inefficient API, a rapidly growing user base and dataset, or a cycle drain attack.

The CycleOps team has been working on a new feature we’re calling anomaly detection that will alert your team when a canister’s cycles balance immediately drops or spikes well outside of its normal cycles usage variability.

Below is “work in progress” video of our work on this feature, showcasing how our anomaly detection algorithm performs against real canister historical cycles data in identifying cycles usage anomalies.

Milestone Preview - Cycles Anomaly Detection

Outside of general feedback on the CycleOps service, we’d like some specific feedback from developers regarding the following questions:

• Is this type of a notification important to you? Why, and are there other canister level notifications that would be important?
• Are email notifications satisfactory? How would you like to receive anomaly notification alerts?
• Do you want to configure the tolerance for anomaly detection based on your specific use case? (i.e. more/less sensitive to alerts?)

[Week_ICP1]

This product will convenient development a lot. Hope to see more features.

[icme]

The CycleOps team is excited to open-source a new cycles-manager library, which provides a simplified, permissioned cycles management framework for Internet Computer applications.

GitHub - CycleOperators/cycles-manager: A library for managing cycles across...

Our hope is that developers are inspired by these patterns to build similar libraries in Rust, TypeScript (Azle), and Python (Kybra). To learn more, head over to the official announcement

Introducing the cycles-manager library

Hello IC Developer Community According to developers, managing cycles is a major pain point Luckily, at CycleOps, we spend all our waking cycles thinking about one thing…cycles That’s why our team is proud to open-source a new cycles-manager library, providing a simplified, permissioned cycles management framework for Internet Computer applications. [X, X Everywhere] Motivation Multi-canister applications tend to have the follo…

[Manu]

hey @icme! Do you have any plans for adding other ways for cycleops to read the cycles balance of canisters? I would be interested in using cycleops to monitor the cycles balance of some NNS-controlled canisters (like the ckBTC canisters), but I don’t think the “balance checker” as controller is a nice way for NNS controlled canisters: now everybody that wants to ensure ckBTC is NNS-controlled would have to additionally check the balance checker canister instead of just seeing that the NNS root is the only controller.

Have you thought about defining some “standard” by which a canister can expose its cycles balance (in application space, so just a standard canister method), and then cycleops could use that as an alternative to read the cycles balance?

[levi]

service cycles-balance-specification : {
    cycles_balance : () -> (nat) query;
}

[icme]

Hey @Manu thanks for your interest in using CycleOps.

First off, I want to say that I 100% understand the hesitation behind adding a controller to any NNS-controlled or other high profile canisters (like ckBTC), and do not advise anyone adding a controller to a canister if they do not either know and trust the controller, or trust the underlying technology with which that controller or and canister is associated.

We started off on the “standards” route, but ran into issues around messaging guarantees, specifically if a malicious canister decides to put up a barrier, meaning that in this case our service canister would no longer be upgradeable.

The biggest problem with the IC - InterCanister Calls

Yes, it is built into the communication model that an inter-canister call is guaranteed to give an answer and indeed this gives a risk you get stuck waiting for a non-responding canister. For this reason the [best practice guidance for secure canister development] (Rust Canister Development Security Best Practices :: Internet Computer) recommends to only talk to trustworthy canisters. This guidance leaves open the question of how to determine a canister is (and remains) trustworthy and excludes…

GitHub - nomeata/ic-barrier: Delay responses on the Internet Computer, for...

Having named callbacks, as described here would allow us to modify our architecture to monitor 3rd party canisters without trust.

@Manu

But now that you have valid (named) callbacks, you can actually make things reliable, eg by retrying in the callback if you get an error back. Note that this allows you to build reliable communication, because there is a guarantee that the callback will always be called for every call.

One potential work around would be to provide a 3rd-party library in Rust, Motoko, Azle, etc. that would allow us to one-shot (fire and forget) into a standardized endpoint API, and then have that endpoint fire back data. At this point though, we realized that we could run into issues since we would no longer be able to trust the protocol to control the data being passed back to us and it could potentially be forged to clog up canister ingress or manipulated to stage a cycle drain attack against our service.

So how did we eventually decide on a blackhole? Last year, I remember having a conversation with @jorgenbuilder and @kpeacock about the sparse use of blackholed canisters in the IC ecosystem, issues with trusting 3rd party canisters, and specific use-cases that require trustless interaction and would make a lot of sense for a blackholed canister.

The blackhole approach we landed on for CycleOps was actually inspired by @PaulLiu’s wonderful ic-blackhole application.

GitHub - ninegua/ic-blackhole: Once a canister sets its only controller to a...

Blackholing our balance checker & leveraging the canister_status API of the IC management canister allows us to use the features and functionalities of the Internet Computer to securely monitor canisters dapps at scale without requiring that developers add a single line of code to their canisters. This monitoring feature that isn’t possible on other clouds or blockchains, so we decided to embrace no-code, IC native monitoring as a superpower of the IC and CycleOps.

Since the blackholed balance checker canister’s code and wasm running will never change, deciding to add it as a controller just requires a one-time code review of this canister. Additionally, the CycleOps monitoring solution allows developer teams to turn monitoring on or off at anytime with a simple shell command instead of requiring a full canister upgrade and the potential risks associated with canister upgrades.

To top it all of, since the CycleOps team trusts the IC and the canister management API, we know that the cycles balance data coming back from the canisters we monitor is legitimate and cannot be forged.

We 100% understand why there would be valid concerns around adding a controller to a canister, which why in addition to black-holing the balance checker, we open sourced its code and have provided steps one can use to verify the code, wasm hash, and controllers of that canister. We’ve also provided a GitHub action for this!
GitHub - CycleOperators/BalanceCheckerVerification: Blackhole and wasm verification of the CycleOps StatusChecker canister.

This all being said, we are in no way entrenched in our current monitoring approach, and would be happy to receive feedback from the DFINITY team regarding safe, trustless ways that we can monitor 3rd party canisters.

[Manu]

Thanks a lot for the detailed answer! I do want to make it clear that I think the approach of a blackholed monitoring canister which is a controller (building on Paul’s blackhole) is a great solution for almost everything and I of course fully trust it works, I just think it’s not a great fit for NNS-controlled things because it makes it harder for people to verify that something is NNS-controlled.

My bad, I forgot about that discussion on upgradability. The named-callbacks are still a planned feature, but we have not made a ton of progress on it yet. The only alternative would be to allow a canister to expose it’s canister status (like was discussed in NNS Proposal: Make canister_status public to anyone).

[lukevoz]

Hi guys @icme @jorgenbuilder @quint , I tried to reach out to you on twitter (cycleops), but seems you don’t see messagess from people you don’t follow and the discord link from your docs does not work either.

I can read in your docs that it could be possible to also manage cycles for dynamically spawned canisters. It is something that I would like to use for our Nostric project.

Could you please give me any directions how can I make this working?

Thanks so much for the reply and your work so far. The tool is great!

[icme]

Hi Lukas,

Sorry about the Twitter and Discord contact issues - they should be fixed now .

Right now, there’s a somewhat complicated way that you can programmatically add and control a canister via CycleOps, but it’s not documented, nor is it a recommended developer experience (although we do our integration testing this way).

If you want to dynamically spin up a large amount of canisters, I recommend using the cycles-manager library (introduced in this post )

Introducing the cycles-manager library

Hello IC Developer Community According to developers, managing cycles is a major pain point Luckily, at CycleOps, we spend all our waking cycles thinking about one thing…cycles That’s why our team is proud to open-source a new cycles-manager library, providing a simplified, permissioned cycles management framework for Internet Computer applications. [X, X Everywhere] Motivation Multi-canister applications tend to have the follo…

That all being said, you aren’t the first person to ask about a “CycleOps” SDK , and I’d be more than happy to hop on a call and learn more about the type of developer experience you’re imagining.

Just sent you a DM!

[icme]

New CycleOps feature alert - Monitoring Overviews

Now, when you log into your CycleOps dashboard, you’ll be able to easily see:

• Total Burn : view the aggregate cycles of all of your canisters over the past month
• Canister Health Summary : see if any of your canisters are healthy , low on cycles , or have frozen

Taking feedback from you!

Many of you currently use CycleOps to monitor multiple canisters , with some of you monitoring over 20 canisters!

Interpreting this much data can be difficult, so over the next few weeks we’ll be rolling out updates that enhance and simplify the monitoring experience.

If you have any feedback or feature requests you can reach out to us on Twitter or our new OpenChat community!

[icme]

New CycleOps feature alert - Improved Canister Dashboards

The CycleOps team is pleased to announce Improved Canister Dashboards , providing memory history , idle cycles burn , module hash , time until freezing threshold hit , and more metrics to help you better understand your canisters and keep them healthy

The full list of additional metrics (in addition to Cycles Balance & Burn Rate) now includes

• Memory (time series)
• Idle Cycles Burn
• Wasm module hash
• Controllers
• Current subnet (for tracking with subnet splitting)
• Estimated time until:
  • Your canister’s top-up threshold is hit (and is topped up with cycles by CycleOps)
  • Your canister’s freezing threshold is hit (if you weren’t using CycleOps and your canister isn’t topped up)

[icme]

New CycleOps Features

Improved Canister Organization and Searchability

Say you’re an IC dev that’s been around the block. This isn’t your first rodeo, and you have tens of canisters you’ve spun up since Genesis

Managing cycles for your canisters is a painful manual process , and you got so many that you need a way to organize your canisters and interpret their data .

Introducing CycleOps Projects and Canister Tags

• Canister Projects allow developers to organize and analyze groupings of canisters belong to a specific project or use case

• Canister tags provide a secondary level of organization when searching through all the canisters you’ve deployed. Is it a frontend canister or backend canister? Ledger or Asset canister? Was the canister written in Motoko, Rust , or Azle?

To top it all off, now when you search by a canister project or tag, your canister overview stats update, allowing you to view just how many cycles that sub-grouping has been burning.

Canister Memory Thresholds and Alerts

Worried about a spike in storage costs or pre-defined canister memory limits?

CycleOps now provides memory notifications with per-canister alerting thresholds , allowing you to get notified via email long before memory becomes a problem

What’s next?

January has been a busy month for the CycleOps team, but we’re not slowing down anytime soon.

Have a canister management or DevOps feature you’d like us to add to our roadmap? Reach out to us on Twitter or OpenChat!

[icme]

New CycleOps Feature

Pay for automated monitoring and canister topups with your own cycles via the new icrc2 enabled cycles ledger

To learn more about this feature, as well as how you can integrate with/migrate to the new cycles ledger,
head over to the feature annoucement post