Candid decode error: "type mismatch: type on the wire rec_4, expect type record ..."

ilbert · October 27, 2023, 5:33pm

In our IC WebSocket JS SDK, we’re trying to make it possible for developers to not care about serialization and deserialization of application messages manually, while at the same enforcing them to implement a strong typing system between their canisters and their clients.
To do so, we require the generated actor in the parameters of the IcWebSocket constructor and extract the developer’s application message type from the ws_message method using this function:

github.com

omnia-network/ic-websocket-sdk-js/blob/e2317b52f6a60200e9fbae97b5874f5d3a8140ea/src/idl.ts#L114-L137


      
          /**
           * Extracts the message type from the canister service definition.
           * 
           * @throws {Error} if the canister does not implement the ws_message method
           * @throws {Error} if the application message type is not optional
           */
          export const extractApplicationMessageIdlFromActor = <T, S extends _WS_CANISTER_SERVICE<T>>(actor: ActorSubclass<S>): IDL.Type<T> => {
            const wsMessageMethod = Actor.interfaceOf(actor)._fields.find((f) => f[0] === "ws_message");
          
            if (!wsMessageMethod) {
              throw new Error("Canister does not implement ws_message method");
            }
          
            if (wsMessageMethod[1].argTypes.length !== 2) {
              throw new Error("ws_message method must have 2 arguments");
            }
          
            const applicationMessageArg = wsMessageMethod[1].argTypes[1] as IDL.OptClass<T>;
            if (!(applicationMessageArg instanceof IDL.OptClass)) {
              throw new Error("Application message type must be optional in the ws_message arguments");

This file has been truncated. show original

Then, we’re using the obtained IDL type to decode the WebSocket message content coming from the canister:

github.com

omnia-network/ic-websocket-sdk-js/blob/e2317b52f6a60200e9fbae97b5874f5d3a8140ea/src/ic-websocket.ts#L483


      
                this.onopen.call(this, new Event("open"))
              } else {
                logger.warn("[onopen] No onopen callback defined");
              }
            }, "Calling onopen callback failed");
          }
          
          private async _callOnMessageCallback(data: Uint8Array): Promise<void> {
            if (this.onmessage) {
              logger.debug("[onmessage] Calling onmessage callback");
              const decoded = IDL.decode([this._applicationMessageIdl], data)[0] as ApplicationMessageType;
          
              await safeExecute(() => {
                this.onmessage!.call(this, new MessageEvent("message", { data: decoded }))
              }, "Calling onmessage callback failed");
          
            } else {
              logger.warn("[onmessage] No onmessage callback defined");
            }
          }

However, if in the tests everything goes fine (using a custom created actor), when deploying it on the example, it gives some errors

the extractApplicationMessageIdlFromActor function throws the error:
```
Application message type must be optional in the ws_message arguments
```
which means that the extracted type is not an instance of OptClass, while instead the generated idlFactory has the correct IDL.Opt type.

As a work around to this error, I’ve replaced the instanceof check with:
```
- if (!(applicationMessageArg instanceof IDL.OptClass)) {
+ if (!applicationMessageArg.name.startsWith("opt")) {
```
Is it robust enough?

after fixing the previous error, the IDL.decode call throws:

Error: type mismatch: type on the wire rec_4, expect type record {text:text; timestamp:nat64}
    at Ba.checkType

(example’s application message type declaration here)

Should we extract the IDL type in a different way? What are we missing?

chenyan · October 27, 2023, 6:32pm

How do you get the IDL type? I suspect both questions are related to a RecClass wrapper in the JS IDL definition. Can you print the console.log of applicationMessageArg and this._applicationMessageIdl?

ilbert · October 27, 2023, 6:44pm

The whole extraction of the IDL type is done in the extractApplicationMessageIdlFromActor.

applicationMessageArg (hex): 4449444c016c02ad99e7e70471d6a9bbae0a7801000470696e673851820ceb0a9217
this._applicationMessageIdl (couldn’t really copy it from browser Dev Console):

image946×934 120 KB

chenyan · October 27, 2023, 7:23pm

How do you construct the input actor then? If https://github.com/omnia-network/ic-websocket-sdk-js/blob/e2317b52f6a60200e9fbae97b5874f5d3a8140ea/src/idl.ts#L136 works, it means applicationMessageArg instanceof IDL.RecClass is true. You can either check applicationMessageArg["_type"] instanceof IDL.OptClass. String matching on the name is probably good enough as well.

For the second question, on https://github.com/omnia-network/ic-websocket-sdk-js/blob/e2317b52f6a60200e9fbae97b5874f5d3a8140ea/src/idl.ts#L136, can you return applicationMessageArg instead of applicationMessageArg["_type"]?

ilbert · October 27, 2023, 8:03pm

The input actor is the actor generated by the dfx generate command and imported from the declarations folder. Here’s the generated idlFactory of used by the actor:

export const idlFactory = ({ IDL }) => {
  const ClientPrincipal = IDL.Principal;
  const ClientKey = IDL.Record({
    'client_principal' : ClientPrincipal,
    'client_nonce' : IDL.Nat64,
  });
  const CanisterWsCloseArguments = IDL.Record({ 'client_key' : ClientKey });
  const CanisterWsCloseResult = IDL.Variant({
    'Ok' : IDL.Null,
    'Err' : IDL.Text,
  });
  const CanisterWsGetMessagesArguments = IDL.Record({ 'nonce' : IDL.Nat64 });
  const CanisterOutputMessage = IDL.Record({
    'key' : IDL.Text,
    'content' : IDL.Vec(IDL.Nat8),
    'client_key' : ClientKey,
  });
  const CanisterOutputCertifiedMessages = IDL.Record({
    'messages' : IDL.Vec(CanisterOutputMessage),
    'cert' : IDL.Vec(IDL.Nat8),
    'tree' : IDL.Vec(IDL.Nat8),
  });
  const CanisterWsGetMessagesResult = IDL.Variant({
    'Ok' : CanisterOutputCertifiedMessages,
    'Err' : IDL.Text,
  });
  const WebsocketMessage = IDL.Record({
    'sequence_num' : IDL.Nat64,
    'content' : IDL.Vec(IDL.Nat8),
    'client_key' : ClientKey,
    'timestamp' : IDL.Nat64,
    'is_service_message' : IDL.Bool,
  });
  const CanisterWsMessageArguments = IDL.Record({ 'msg' : WebsocketMessage });
  const AppMessage = IDL.Record({ 'text' : IDL.Text, 'timestamp' : IDL.Nat64 });
  const CanisterWsMessageResult = IDL.Variant({
    'Ok' : IDL.Null,
    'Err' : IDL.Text,
  });
  const CanisterWsOpenArguments = IDL.Record({ 'client_nonce' : IDL.Nat64 });
  const CanisterWsOpenResult = IDL.Variant({
    'Ok' : IDL.Null,
    'Err' : IDL.Text,
  });
  return IDL.Service({
    'ws_close' : IDL.Func(
        [CanisterWsCloseArguments],
        [CanisterWsCloseResult],
        [],
      ),
    'ws_get_messages' : IDL.Func(
        [CanisterWsGetMessagesArguments],
        [CanisterWsGetMessagesResult],
        ['query'],
      ),
    'ws_message' : IDL.Func(
        [CanisterWsMessageArguments, IDL.Opt(AppMessage)],
        [CanisterWsMessageResult],
        [],
      ),
    'ws_open' : IDL.Func([CanisterWsOpenArguments], [CanisterWsOpenResult], []),
  });
};
export const init = ({ IDL }) => { return []; };

Here it is:

Also, I didn’t find a clear way to extract the type of an OptClass except from accessing the protected _type field.

chenyan · October 27, 2023, 10:49pm

I mean I assume this._applicationMessageIdl comes from extractApplicationMessageIdlFromActor. Can you return applicationMessageArg instead of applicationMessageArg["_type"] in extractApplicationMessageIdlFromActor?

ilbert · October 28, 2023, 7:25am

Yes exactly.

Why should I return the Opt type? The encoding and decoding is done using the inner type of the option, because the canister sends the encoded app message through WebSocket, not the option of it (see the code in the example). Anyway, after trying what you suggested, I’m getting the same error:

Error: type mismatch: type on the wire rec_4, expect type opt record {text:text; timestamp:nat64}

chenyan · November 2, 2023, 1:47am

Would be good if you can have a minimal repo that I can try locally.

But before that, let’s try changing this line https://github.com/omnia-network/ic-websocket-sdk-js/blob/e2317b52f6a60200e9fbae97b5874f5d3a8140ea/src/ic-websocket.ts#L483 into

const decodedType = Rec();
decodedType.fill(this._applicationMessageIdl);
const decoded = IDL.decode([decodedType], data)[0] as ApplicationMessageType;

ilbert · November 2, 2023, 9:29am

chenyan:

But before that, let’s try changing this line https://github.com/omnia-network/ic-websocket-sdk-js/blob/e2317b52f6a60200e9fbae97b5874f5d3a8140ea/src/ic-websocket.ts#L483 into
const decodedType = Rec();
decodedType.fill(this._applicationMessageIdl);
const decoded = IDL.decode([decodedType], data)[0] as ApplicationMessageType;

I’m getting the same error, but with rec_5 instead of rec_4:

Error: type mismatch: type on the wire rec_5, expect type record {text:text; timestamp:nat64}

Other information that could help:

I’ve also tried to use a variant type as the application message, but I’m getting the exact same mismatch error
```
Error: type mismatch: type on the wire rec_4, expect type variant ...
```
if I use a plain text type it works smoothly (and I believe it’s the same for all other primitive types).

I think it has somehow to do with the JS compiler that I’m using (Vite+Rollup).

Regarding reproducing it, it’s a bit hard to create a minimal repo since there are many dependencies involved (Rust CDK (which version is still not published), JS SDK (which version is still not published) and the WS Gateway that has to run separately). I’ll see what I can do.

ilbert · November 2, 2023, 10:18am

Diving into the @dfinity/candid source code, I can’t understand how is it possible that I’m falling in the throw case:

github.com

dfinity/agent-js/blob/83307737c4088e923edbdaeaeb5040dc7e7f4282/packages/candid/src/idl.ts#L246


      
          
          export abstract class ConstructType<T = any> extends Type<T> {
            public checkType(t: Type): ConstructType<T> {
              if (t instanceof RecClass) {
                const ty = t.getType();
                if (typeof ty === 'undefined') {
                  throw new Error('type mismatch with uninitialized type');
                }
                return ty;
              }
              throw new Error(`type mismatch: type on the wire ${t.name}, expect type ${this.name}`);
            }
            public encodeType(typeTable: TypeTable) {
              return typeTable.indexOf(this.name);
            }
          }
          
          /**
           * Represents an IDL Empty, a type which has no inhabitants.
           * Since no values exist for this type, it cannot be serialised or deserialised.
           * Result types like `Result<Text, Empty>` should always succeed.

because the following condition is not met:

github.com

dfinity/agent-js/blob/83307737c4088e923edbdaeaeb5040dc7e7f4282/packages/candid/src/idl.ts#L239


      
              return t;
            }
            public _buildTypeTableImpl(typeTable: TypeTable): void {
              // No type table encoding for Primitive types.
              return;
            }
          }
          
          export abstract class ConstructType<T = any> extends Type<T> {
            public checkType(t: Type): ConstructType<T> {
              if (t instanceof RecClass) {
                const ty = t.getType();
                if (typeof ty === 'undefined') {
                  throw new Error('type mismatch with uninitialized type');
                }
                return ty;
              }
              throw new Error(`type mismatch: type on the wire ${t.name}, expect type ${this.name}`);
            }
            public encodeType(typeTable: TypeTable) {
              return typeTable.indexOf(this.name);

while the only case in which the t.name getter returns rec_... is exactly in the RecClass:

github.com

dfinity/agent-js/blob/83307737c4088e923edbdaeaeb5040dc7e7f4282/packages/candid/src/idl.ts#L1318


      
          }
          
          public decodeValue(b: Pipe, t: Type) {
            if (!this._type) {
              throw Error('Recursive type uninitialized.');
            }
            return this._type.decodeValue(b, t);
          }
          
          get name() {
            return `rec_${this._id}`;
          }
          
          public display() {
            if (!this._type) {
              throw Error('Recursive type uninitialized.');
            }
            return `μ${this.name}.${this._type.name}`;
          }
          
          public valueToString(x: T) {

It seems that the compiler is somehow messing up these class implementations and this breaks the instanceof operator. And it seems that for the same reason, the OptClass condition also fails:

Possibly related: Issue with `instanceof` operator · Issue #3333 · evanw/esbuild · GitHub

ilbert · November 2, 2023, 10:56am

I managed to solve the issue!

Based on this comment, I noticed that I was having multiple node_modules/@dfinity/... folders: one in node_modules/@dfinity/* and the other in src/ic_websocket_example_frontend/node_modules/@dfinity/*. The bundler was importing both of them and that’s why the instanceof operator was failing…
After deleting the second one, it worked!

Also happy that the @dfinity/candid and ic-websocket-js packages didn’t have bugs at the end.

Topic		Replies	Views
IC WebSocket - decode incoming message from Candid bytes Rust websocket , ic-websocket	3	323	October 18, 2023
`ic-test-state-machine` failed to decode response as candid type Developers Discussing	5	321	June 25, 2023
Candid failed to decode for internal error Developers candid , Discussing	1	418	June 25, 2023
Convert Candid to JSON Rust candid , rust	27	986	March 13, 2024
Candid field names instead of hashes Developers	6	440	July 27, 2022

Candid decode error: "type mismatch: type on the wire rec_4, expect type record ..."

Related Topics