Technical Working Group: Inter-canister Event Utility Working Group

General
1

Hello everyone,

I’d like to announce a new technical working group :

Topic: Inter-canister Event Utility
DFINITY lead: tbd
Coordinator: @skilesare
Community Coordinator: @ilialor
Discord: inter-canister-events
Meeting Notes: Link
Cadence: Weekly on Wednesday at 7:00pm CET
First meeting: April 3, 2024 at 7:00pm CET
The event is part of the regular calendar for technical working groups . (meeting link will be there soon)

I’m happy to announce the Inter-canister Event Utility working group.

A number of grantees have been working on various forms of Pub-Sub event systems and we decided it was time for all of us to work together.

Many blockchains have event systems. Ethereum EVMs allow contracts to emit events, but not react to them in real-time. On the IC we can have the best of both worlds with a bit of cooperation, technical architecture, and elbow grease. @ilialor, @Carsten, and @skilesare have been working on various forms of even systems for some time now. We’ve all come together and decided that sure does make sense for us all to get on the same page about standards(at least), architecture, and long-term economics(perhaps).

We have the opportunity to add a fundamentally new programming model to the IC by adding pub/sub events. There are challenges. Events need to scale, they need to pay for themselves, and they need to get across subnets. There are a lot of challenges. But they are also scaleable. We’re going to start small and see where we get.

The short-term goals of the group will be to agree on a set of ICRC standards around pub-sub. If we can all agree on some basic standards then we can build libraries that should work for everything from single-canister programming models to small private pub/sub networks to big hairy multi-subnet pub-sub.

Once the standards are agreed on we should be able to build client libraries in motoko, rust, azel, etc that let Canisters listen to reliable broadcast messages traveling from broadcasters to unknown listeners that may be interested in the actions and state changes on all of our dapps.

Along the way, we may even find a way to fix the ‘notify’ problem for ledgers or perhaps give EVMs active pubsub with the IC acting as the web3 glue.

You all are invited to join and contribute.

21 Likes
2

This meeting is on Wednesdays. Today is our first public meeting.

5 Likes
3

Thank you @skilesare for coordinating this working group.

The pace of progress in this ecosystem is fascinating!

2 Likes
4

Will you post summaries of the meetings?

I am very interested in this topic, but unable to make the meetings, most of the time.

1 Like
5

I’ll do my best…looking into some tools, to do it automagically:

Notes from the last meeting here: ICEventsWG/Meetings/20240403/Notes.md at main · icdevs/ICEventsWG · GitHub

DFINITY’s Internet Computer: Intern Canister Events Working Group Meeting Notes

Meeting Details

  • Date: [Insert Date]
  • Participants: Austin, Ilia, Sam, Matthew, H Joseph, Ebin, Byron Becker, Ilia, Aga, Zhenya Usenco, Dclan, Nnadozi, Unicorn

Meeting Notes

  • Introductions:

    • All participants introduced themselves at the beginning of the meeting.

  • Discussion on Issue 1 for ICRC:

    • Generic Data Types Proposal:

      • Discussed implementing generic data types with an example structure { owner: principal; subaccount: blob; }.

    • Public Event Publishing/Subscribing:

      • Highlighted the advantage of using pub/sub and events, emphasizing the ease of interoperability without the need for coordination.
      • Proposed a GenericEvent and DataSchema structure to facilitate this process, including event filters, publisher information, schema details, and data expiry.

    • Value Type and Metadata:

      • Debated on the use of just a blob versus the Value type from ICRC3 for metadata, pointing out the necessity for a strict definition and hash calculation.
      • Mentioned ICRC16 as a superset of Value, referencing external resources for further details (ZhenyaUsenko’s GitHub).

    • Client and Methods:

      • Discussed keeping the methods and client implementations generic but useful and interesting, focusing on minimalism.

  • Best Practices:

    • Advised that event data should remain small for efficiency.
    • Byron Becker suggested starting with different use cases and engaging with projects to understand their needs and data requirements.

  • Design Considerations:

    • The design for an event pub-sub system should account for the data’s size and schema, including steady streams versus spikes, broad versus specific use cases, and achieving high throughput through schema constraints and optimization.
    • Mentioned examples from Web 2.0, such as segments and other event publishing frameworks like Kafka.

Next Actions:

  • Byron: To research Web2 messaging/event frameworks.
  • Everyone:
    • Bring a proposal for the next meeting focusing on external actor endpoints and internal data structure/workflow.
    • Review the pros and cons of ICRC-16 versus ICRC-3 values.
  • Austin: Develop a more descriptive solution for token tracking.
  • Matt: Explore multiplayer capabilities with pubsub, including potential support for websockets and WebRTC.

Additional Notes:

  • The group emphasized collaboration and open discussion, with a focus on leveraging DFINITY’s Internet Computer capabilities to enhance interoperability and efficiency in event publishing and subscription models.
  • There was a consensus on the importance of practicality and minimalism in designing the system, ensuring it remains both powerful and user-friendly for various use cases.
5 Likes
6

The agenda for tomorrow’s Event Working Group is posted over at GITHUB.COM If you would like to add something, or are on the list and can’t make it, let me know.

3 Likes
7

I would propose NATS as an excellent reference architecture for a decentralised high performance “web2” messaging platform. It is a fully open source project evolving since 2011 and is widely used. It’s Core NATS server is implemented in Go and is a pure pub/sub messaging model with a simple (word) token based subject(topic) used to identify/filter/route messages and a generic byte_array payload. Authentication is decentralised and client controlled using pub/priv keys and JWTs for authorisation claims. Although it would not fit directly into the IC canister event system requirements, I believe there Is a lot to learn from NATS that we could use here.

Ideally we would reach some level compatibility between the IC canister event on-chain messaging and a off-chain decentralised messaging system like NATS to solve a range of architectural integration problem likely to appear as more web2 development teams seek to leverage the IC

1 Like
8

Are there any thoughts on adopting CBOR as a standard binary encoding & encapsulation format for message payloads? I know it is used already in a number of IC code libraries & standards.

Pros would be
(a) wide language support incl Rust, Motoko(?), js, python for canisters and just about any other used for off-chain clients, also many developer tools support inspection;
(b) easy to encode JSON data into compact and canonical form for transport (so deterministic data structure removing any text formatting variances of same data content);
(c) 1-4 byte Semantic Tags used in front of encoded binary data structures which can be used to either completely embed schema information or as a very compact reference to an external schema via open extension to the CBOR standard semantic tags dictionary (e.g. we could formally assign a 2byte CBOR semantic tag for each ICRC-X defined data schemas)
Also there is an assigned 1byte tag for “CBOR data” which can be used as a ‘magic number’ for the first byte of a message payload to distinguish it trivially from a text payload or other binary message format.

Any thoughts on cons to this approach?

3 Likes
9

Awesome example. I’ll add it to the list.

10

The actual messages will be going across the wire in candid because that is how the IC works and we’re generally targeting either ICRC16 or the Value type from ICRC3, but one of the varients is #Blob, so you could easily put a cbor message in there.

That being said, CBOR is awful to work with in Motoko.:joy:. So unless we seriously level up our parser game you’d be ruling out any motoko-based services easily integrating.

The main issue is that Motoko doesn’t have reflection, so each publisher would need to craft a specific cbor-based parser for there events for them to be converted into candid types(and even then, the candid types would require syncing between publisher and subscriber.

The nice thing about a candid message in ICRC16 or value is that it is reflective without a bunch of byte reading and parsing. This may have poor performance consequences…we’ll need to test.

1 Like
11

Just for fun Ill link my previous project that I put on hold till something like this would come around.
Essentially building an RSS like feed/bridging Web2 RSS over to the IC.
Probably doesn’t have a lot of useful things for this conversation, but probably could play with implementing the ICRC-72 into this as a proof of concept

2 Likes
12

Please let me know if I missed anything:

Event Working Group Meeting Notes

Date: April 10, 2024

Attendees:

  • Ilia Agafonov
  • Austin Fatheree
  • Ethan Celletti
  • Byron Becker
  • Lachlan Witham (Icarus)
  • Jorge Costa
  • Gilles Pirio
  • Sam DRISSI

Introductions:

  • Ethan Celletti: Working on an RSS pubsub, C# agent, Motoko, and a game on the IC (DAOBall).
  • Lachlan Witham (Icarus): From Melbourne, AU. Gen2 Node Provider. Part of the decentralized AI Working Group, interested in infrastructure and keen to listen in.
  • Jorge Costa: Has been developing for two months, observing. Software developer for 6 years, with experience in AI, contracts for EVM.
  • Gilles Pirio: Based in San Francisco. Former DFINITY employee, here to learn.

Discussion Points:

  • Naming Conventions:

    • Ilia highlighted the importance of including “event” in the standard for clarity.

  • Identifying Producers and Consumers:

    • Byron Becker discussed the potential need for two identifiers to distinguish between producers and consumers. He suggested using Nats as ULIDs for this purpose.

  • ID Systems:

    • Gilles Pirio recommended looking into RabbitMQ for a good ID system, emphasizing its effectiveness.

  • Publication Modes:

    • Byron raised a question about publication modes, differentiating between wanting the latest data versus wanting the entire history.
    • Gilles questioned the necessity of data history.

  • History and Pub/Sub Separation:

    • Austin suggested possibly splitting history and pubsub into two parts: a basic ICRC-72 and an extension ICRC.

  • Data Encoding:

    • Icarus brought up CBOR, noting its expressiveness and efficiency, and mentioned creating a GitHub issue regarding it. However, Austin noted the difficulty of working with CBOR in Motoko.

  • Error Handling and Resending:

    • Gilles inquired about error handling and the possibility of resending data similar to RabbitMQ, indicating different application modes.

  • Standard Flexibility:

    • Austin emphasized that the standard should support a wide range of operations and applications, from simple single canister pub/sub models to massive IC-wide tools, with the publish method remaining consistent across various expectations and system specifics.

Action Items:

  • Austin Fatheree:
    • To create an open chat room.
    • To collect open chat and principals, and create a GitHub issue accordingly.
  • Working Group:
    • To address GitHub issues asynchronously.
    • To set the notification bell on the Working Group GitHub.

Next Meeting:

  • Scheduled for next week.

Additional Notes:

  • The meeting focused on addressing technical aspects of event publishing and subscription on the IC, including naming conventions, identifying producers and consumers, and discussion on publication modes.
  • Participants brought a diverse set of experiences and backgrounds, enriching the discussion with different perspectives on how to improve the event working group’s standards and practices.
  • The action items set a clear path forward for the group, emphasizing collaboration and the efficient use of GitHub for asynchronous work.
2 Likes
13

Please see this github issue for submitting your principal and your oc user name: Action Item: Please provide a principal(see instructions) and oc name · Issue #10 · icdevs/ICEventsWG · GitHub

1 Like
14

The OpenChat group is set up. It is a public group for now, so I don’t need individual oc usernames…just join up:

15

I just want to do a fly-by post and urge the working group to use as few IC-specific concepts and APIs as possible. For example, I would suggest embracing JSON or any other format over Candid, using HTTP or other protocols that already exist as applicable, etc.

If other pub/sub systems already exist, work well, and have major adoption in the traditional software development world, I would consider how we can simply embrace and incorporate those directly.

At Demergent Labs we have finally in the last 3-6 months come around to fully embracing HTTP, JSON, sockets, etc in an effort to remove as much ICP complexity as possible for newcomers.

I urge you do try and do the same.

1 Like
16

The system we’re considering at the moment is currently focused on all inter-canister calls which require candid in order to not splinter the ecosystem. When we get to the external interface there is lots to consider, but my current thinking is that ingress into the system would take place via canister wallet/wapps/whatever you want to call them so as to keep things simple. Once ingressed you’d relay via the maximally interoperable standards.

The current data structure for events is a generic Value/ICRC16 structure which includes #Text and #Blob which an application can of course stick whatever they want into and CBOR has already been discussed. But as noted in our last meeting, using CBOR will likely seriously complicate interoperability.

2 Likes
17

Today’s Agenda: ICEventsWG/Meetings/20240417 at main · icdevs/ICEventsWG · GitHub

18

Event Working Group Meeting Notes

Date: 20240417

Attendees:

  • Austin Fatheree
  • Ilia Agafonov
  • Ethan Celletti
  • Matt
  • Lachlan Witham
  • Zhenya Usenko

Agenda and Discussion Points:

  1. Token Vote:

    • The group agreed to ratify the first token mint based on the discussion in GitHub Issue #12.

  2. Formalize WG Member List and Role Assignment:

    • Decision: Formal WG member list will be used. Members can join by asking, and roles will be assigned liberally unless there is a consensus to limit a role.
    • Specific Roles Discussed:
      • Austin Fatheree: Thought Leader, Voting Member
        • Assignment: Create weekly meeting agenda.
      • Roles Defined:
        • Thought Leader: Review issues, provide proactive responses, and insight.
        • Voting Member: Vote on “Vote Required” GitHub issues.
        • Secretary: Responsible for recording meetings, producing notes, and providing attendance.
    • Austin will brainstorm a more formal list of roles.

  3. Issue Review:

    • Issue #2 - Two Types of Subscriber Approach:

      • General consensus reached; raised by Ilia. Discussion focused on systems requiring trusted responses, especially in smaller systems or those dealing only with trusted canisters.

      • Austin asked about the type of response, leading to a decision on using open metadata so systems can implement ICRC16/Value, ensuring that the response from trusted endpoints should only include metadata, not rich stateful data.

      • We have general consensus.

  icrcx_event_listener : (event: Event) -> async (); //untrusted
  icrcx_event_listener_trusted : (event: Event) -> async (opt Value);  //#Map for maps, #Nat for ID.

Presupposition: The trusted return type should only include response metadata and typically not have rich, stateful data.(emit your own event if you want to talk back to a publisher).

  1. Issue #3 - Message Identifier Discussion:
    • Raised by Austin, discussion on whether to use variants for message identifiers and the necessity of size checking if using a ‘nat’.
    • Consensus on broadcaster needing to validate message size.
    • Group ready to vote on at least using nats for message identifiers, and Previous Message optional.
    • Notification ID will be moved to a different Issue
  2. Ilia’s Pull Request:
    • Update on the name in the draft related to event identifiers. (Make sure it has Events in it … see Pull Request 1).

Action Items:

  • Austin Fatheree: Create Votes for Issue 2 and 3
  • Austin Fatheree: Create Role list
  • Group: Continue to create new issues and address existing issues
2 Likes
19

The agenda for tomorrow’s meeting is up: ICEventsWG/Meetings/20240424/agenda.md at main · icdevs/ICEventsWG · GitHub