Breaking Registry Changes for Large Records

Developers
1

You can stop reading now if you do not call the following methods of the Registry canister:

  • get_certified_changes_since
  • get_changes_since
  • get_value

Dear most esteemed Registry clients,

TL;DR: DFINITY proposes to make some breaking changes to the aforementioned methods. That is, affected clients will need to make changes, but a gradual migration path is planned.

The problem with the current interface is that messages must not exceed ICP’s 2 MiB size limit. Without these changes, the upcoming vetKeys feature would exceed that limit.

Very briefly, what clients will need to do when fetching a larger record is to download it over multiple calls. Requests would look exactly as they do now. The difference is that clients will need to support a slightly enhanced response. This is only needed if the record is large. If a client is already using a DFINITY-developed Registry library, all they would need to do is upgrade this library.

The exact details of how to migrate are explained in this .proto file, but to slightly elaborate, clients would need to switch from decoding to new “high-capacity” types, which contain an additional large_value_chunk_keys field. Using this field, clients can make calls to a new get_chunk method, thus spreading the large record over multiple calls.

Callers CAN safely switch right now. Such changes will be NEEDED later, when large entries are added to the Registry. It is planned that large entries would be added some time in the second quarter of 2025. The exact date will be announced as the time approaches.

Please, check out the detailed plan in the aforementioned .proto file. If you have concerns, please, reach out. Nobody likes breaking changes, but in order to support larger records, this seems about as minimally invasive and gradual as one can imagine.

As always, your continued support of the Internet Computer Protocol is greatly appreciated. Happy smart contracting!

Sincerely,
Daniel Wong

5 Likes
2

Thanks for the update Daniel. With these changes can you additionally provide candid endpoints for these APIs so they can be more easily consumed by canisters?

3

This x 100 and paginated

4

Thank you for the feedback; it makes sense. It’s not in scope for this project, but maybe we can do it as follow up.

FWIW, while I was designing this, I didn’t see any reason why these must not be available in Candid as well. IOW, we have nothing against what you are asking for. I think it was done this way originally because at the time, Candid was not mature enough.

UPDATE: I thought about this later, and realized it would be a bit harder to come up a PURE Candid version of the get_certified_changes_since method, because even though at the top level it would probably be pretty easy to transcribe from Protocol Buffers to Candid, the response contains blobs that are PB-encoded, and those cannot simply be transcribed to Candid, because the certification relies on the specific bytes in those blobs, not their decoded forms :face_exhaling: However, for the non-certified methods (e.g. the non-certified get_changes_since method), this isn’t a problem; we could just transcribe. Anyway, DFINITY still has no concrete plan to add Candid equivalents, so if you need this data (soon), please, consider using the Protocol Buffers API, such as it is.

1 Like
5

A version of the nns_data_provider library that supports chunked responses is available now in the master branch of the ic repo. Therefore, now would be a good time for users of that library to upgrade. This should not require any client code changes (unless of course, I am forgetting something).

The Registry canister has the code needed to start accepting large mutations, but this feature has not been enabled (yet), so there is still time to for clients to migrate (e.g. by simply doing the nns_data_provider library upgrade mentioned in the first paragraph). We would like to enable larger mutations in the Registry canister in a couple of weeks or so. I will update this thread as the exact time becomes clearer.

6

Registry was upgraded on Monday to support large values (via proposal 136988). No large values have been created yet, but that is scheduled to happen on Monday.