Mismatch between ICRC-3 standard and Dfinity ICRC ledger implementations: options for moving forward

Developers
,
1

Problem

The ICRC-3 standard specifies a certain format for the certificate returned from the icrc3_get_tip_certificate() endpoint. According to this standard, these certificates should contain:

  1. A last_block_index path, leading to a node where the value is a LEB128-encoded integer.
  2. A last_block_hash path, leading to a node where the value is a 32-byte blob.

This format, hereafter referred to as ICRC-3 certificates, is implemented by the cycles ledger, which follows the correct paths but uses an incorrect encoding for the last_block_index. Specifically, the cycles ledger encodes last_block_index as a 8-byte big endian encoded u64 byte array rather than a LEB128 integer, which diverges from the ICRC-3 specification.

Furthermore, other ICRC ledgers (SNS and ck ledgers) implement certificates with:

  1. A last_block_index path, leading to a node where the value is an integer encoded via to_bytes.
  2. A tip_hash path, pointing to a 32-byte blob.

These certificates, which we’ll refer to as legacy certificates, are used by Rosetta nodes (through the get_blocks interface) and likely other external clients that rely on this legacy format.

Main Concern: Several ledgers currently claim compatibility with ICRC-3, but their certificates do not actually conform to the ICRC-3 standard. This discrepancy means that these ledgers are not fully compatible with the ICRC-3 specification; aligning them with ICRC-3 would require substantial changes that may disrupt clients already dependent on the legacy format.

Additional considerations:

  1. Compatibility: The icrc3_get_tip_certificate endpoint isn’t actively used in production, meaning no current clients rely on ICRC-3 certificates in real time. However, this will change and a consistent format is needed across ledgers to ensure smooth integration for future clients.
  2. Backward Compatibility: Extending certificates without disrupting current clients is feasible only if new data is added under unique paths. Modifying the values on existing paths, like the value of last_block_index, would force changes to the certificate’s type and could impact legacy clients.

Options

Option 1: Migrate Ledgers to Match the ICRC-3 Standard

  • Required Changes: Upgrade ledgers to produce ICRC-3 certificates with the correct encoding (LEB128) on last_block_index, ensuring current clients can interpret both formats temporarily, then eventually transition entirely.
  • Pros:
    • Standard Consistency: Maintains the integrity of the ICRC-3 standard without alteration, avoiding a precedent of changing community-voted standards.
    • Long-term Uniformity: Establishes a single certificate format across all ICRC ledgers, simplifying future client interactions.
  • Cons:
    • Client Breakage Risk: This approach depends on all clients upgrading to support both certificate formats temporarily, risking potential breakage for clients that fail to update in time.
    • Complex Coordination: Requires coordination with all existing clients (e.g., Rosetta nodes, external applications) to implement updates, involving substantial communication and possibly out-of-band upgrades with some consumers of these certificates like the Taurus integration.
    • Unknown Clients: The number of unknown clients relying on the legacy format is uncertain, adding further risk.

Option 2: Modify the Standard to Align with Legacy Format

  • Required Changes: Revise the ICRC-3 standard to recognize only the legacy format as compliant. The cycles ledger would also need adjustments to align with this updated spec.
  • Pros:
    • Immediate Compatibility: Avoids disrupting existing clients that rely on legacy certificates, eliminating the need for client-side updates or synchronizations.
    • Minimal Client Impact: Since no clients currently use the icrc3_get_tip_certificate endpoint, modifying the standard to support the legacy format sidesteps immediate integration concerns.
  • Cons:
    • Altering Standards: Changing an agreed-upon standard post-vote could weaken trust in community governance and future standards.
    • Potential Confusion for New Integrations: If new integrators expect the original ICRC-3 format, this change could lead to confusion or compatibility issues down the line.

Option 3: Introduce Backward Compatibility in the Standard

  • Required Changes: Update the ICRC-3 standard to accept both certificate formats.
  • Pros:
    • Flexible for Existing and New Clients: Accommodates legacy clients without requiring immediate updates, while still allowing new clients to use the original ICRC-3 format.
    • Maintains Standard Continuity: Adjusting the standard in this way acknowledges existing usage while preserving a broader implementation path.
  • Cons:
    • Added Complexity for New Clients: New integrators must implement logic to detect and support two certificate formats, which increases complexity and reduces development efficiency.
    • Client Dependence on Ledger Instances: Existing clients become tied to specific ledger implementations; if they interact with ICRC-3-compliant ledgers, they must handle both certificate types, complicating integration.
    • Extending the Standard: The standard would need to be extended to accommodate both legacy and ICRC-3 certificate formats, which could lead to long-term fragmentation. This extension might increase the risk of future inconsistencies, add to the maintenance burden, and complicate the onboarding of new clients who would need to navigate both formats rather than a single unified one.
    • Modifying a Voted-on Standard: Changing an already community-voted standard to accommodate legacy implementations could set a precedent that weakens governance and raises concerns about stability in community-adopted standards.

Option 4: Develop a New Standard & Deprecate ICRC-3

  • Required Changes: Introduce a new standard with unambiguous encoding and path requirements, while deprecating ICRC-3.
  • Pros:
    • Cleaner Future State: A new standard allows for a fresh implementation, reducing ambiguity in certificate formats and avoiding dependencies on legacy implementations.
    • No Immediate Impact on Current Clients: Existing clients can continue to rely on the legacy format while new clients transition to the updated standard, minimizing disruptions.
  • Cons:
    • Extended Timeline: Introducing a new standard and going through a deprecation process could be time-consuming due to the necessary community vote and the creation of a deprecation mechanism.
    • Inaugural Deprecation: This would set a precedent for deprecating a standard within the ICRC, potentially complicating governance and creating uncertainty for future standards.
    • Disruption from Deprecation: ICRC-3 has already gained some recognition and is somewhat known within the community. Deprecating it may create confusion and disruptions, especially for clients and developers familiar with the ICRC-3 standard.

Call for community feedback

We invite the community to weigh in on the proposed options. In particular:

1. Other Implementations: Let us know if there are any other ledger implementations that fully follow the ICRC-3 standard, as this could affect the feasibility of each option.
2. Client Impact: For those actively using these ledgers, what level of disruption would each option cause?
3. Trade-Offs in Agility vs. Stability: This is not the first time where standards and already deployed canisters diverge, and aligning the two is usually challenging. Share your preference on the balance between being pragmatic and modifying an existing standard (considering that the ICRC-3 standard is still new and not widely used) versus being more principled and deal with the challenge of preserving the standard as-is and adapting clients to support a smooth transition to a consistent state.
4. Alternative Solutions: If you have suggestions for other ways to address this issue, please share them so we can consider all possible paths forward.

We will discuss this topic in an upcoming session of the working group, but please share your insights and suggestions to help us navigate these options.

2

I’m not sure this is true. In addition, the Subscription utility that DFINITY will reject any upgrades to via the NNS uses the currently defined spec and can’t be upgraded(https://dashboard.internetcomputer.org/canister/fe5iu-uiaaa-aaaal-ajxea-cai). There are likely other ledgers that are blackholed that use the motoko ICRC3 library that use this encoding(@corntoshi?)

Option 5:

  • Update clients to check to_bytes encoding and 8-byte big-endian encoded u64 byte array encoding and not choke.
  • Add last_block_hash to SNS and ckLedgers. Update known clients to use this, but leave tip_hash there for legacy.
  • If a new standard is needed, make it a client standard that is more flexible on the last_block_index encoding and name the cycle ledger by name if it will be the only incompatible ledger.

I understand that writing an extra leaf to the SNS ledgers is a pain, but by doing that you only have the cycle_ledger that is out of sync. That seems easy enough to handle in the known clients. Don’t we need an ICRC-3 Rosetta that uses these endpoints anyway? I don’t think the current rosetta can be used with non-SNS ledgers for some(maybe this) reason.

3

Thanks @bogwar for the summary, and @rupansh-gob for finding it.

The CYCLES-BANK by the CTS implements the ICRC-3 standard as it’s currenly written (method link). We can upgrade it with whatever the outcome of this post will be, we have not built any tools that use that part of the standard yet.

I think the best way is to update the standard with the icrc3_* prefix on the labels in the certificate. I think it is what the working group would’ve done if we didn’t overlook it. The ledgers can be upgraded to add the icrc3_* labels without removing their current labels.

Option-3 can also work.

I think since the standard is new, no one found this difference until now, and this part of the standard is not even available for inter-canister-calls, it is a minimal change that is better taken care of fast and simple.

4

Update on the Path Forward

We’ve decided to move forward with Option 1, as it is the least invasive approach and does not require any changes to the standard. Our plan is as follows:

  1. Update the Cycles Ledger Canister:
    The ledger will be updated to issue ICRC-3 compliant certificates. Since certificates have not been requested from this ledger so far, we anticipate no impact on existing clients.
  2. Engage with Key Stakeholders:
    We will notify known parties using direct integrations about the upcoming breaking change and propose mitigation strategies (see below for the specific change in the case of Rosetta nodes).
  3. Enhance the Rosetta Node:
    The Rosetta node will be updated to support both the existing and ICRC-3 compliant certificates. Clients can distinguish between the two formats by examining the certificate structure:
  • Legacy certificates include a path labeled tip_hash.
  • ICRC-3 compliant certificates include a path labeled last_block_hash .
    Clients should handle both formats to ensure compatibility during the transition period.
  1. Coordinate with Rosetta Users:
    We will notify known Rosetta users about the new release and work with them to ensure a smooth migration to the updated version.
  2. Update the ICRC Ledger Suite:
    Finally, we will bring the full ICRC ledger suite into compliance with the ICRC-3 specification.
5

Discussion thread for proposal 134202 to upgrade the cycles-ledger according to Option 1 outlined above.

1 Like