Using @dfinity/agent in node.js

So I’ve been trying to run queries from node.js, and wanted to document the process, if anyone is interested. I’m also hoping some of the workarounds that I found can be properly patched so we can use services like heroku without having to jump trough hoops.

:~/work/dfinity/node_test_2$ node -v
v10.19.0

:~/work/dfinity/node_test_2$ npm -v
6.14.4

:~/work/dfinity/node_test_2$ npm init

Following the readme from here, I installed the agent package.

npm i --save @dfinity/agent

I created a file app.js, with a simple import, and ran it with
node app.js

const actor = require("@dfinity/agent");

First series of errors seem to stem from the fact that npm doesn’t import other packages. I had to add them manually.

Error: Cannot find module '@dfinity/principal'
#fix:
npm i --save @dfinity/principal

Error: Cannot find module '@dfinity/candid'
#fix:
npm i --save @dfinity/candid

#will need later:
npm i --save node-fetch

With all importing errors fixed, I ran node app.js again, and got the following error:

ReferenceError: TextEncoder is not defined
    at Object.<anonymous> (~/work/dfinity/node_test_2/node_modules/@dfinity/agent/lib/cjs/auth.js:19:50)

This can be fixed by patching the auth.js file, adding the following line after use strict:

global.TextEncoder = require("util").TextEncoder;

With the file patched, we can now import the agent without any errors. Now let’s build an actor for a local deployment, and make a call to a local canister:

app.js:

const Actor = require("@dfinity/agent").Actor;
const HttpAgent = require("@dfinity/agent").HttpAgent;

//a mix of greet, and whoami capsules, with another greetq (as query) added
const idlFactory = ({ IDL }) => {
    return IDL.Service({
        'greet': IDL.Func([IDL.Text], [IDL.Text], []),
        'greetq': IDL.Func([IDL.Text], [IDL.Text], ['query']),
        'whoami': IDL.Func([], [IDL.Principal], ['query']),
    });
};

//replace with any canister id
const canisterId = "rrkah-fqaaa-aaaaa-aaaaq-cai";

const agent = new HttpAgent({
    host: "http://localhost:8000",
});
//needed for update calls on local dev env, shouldn't be used in production!
agent.fetchRootKey();

const actor = Actor.createActor(idlFactory, {
    agent,
    canisterId,
});


let callCanister = async (message) => {
    res = await actor.greet(message).catch(e => { return "Error" + e });
    return res;
}

callCanister("TEST").then(res => { console.log(res) });

Running that code gives another error:

TypeError: Cannot read property 'bind' of undefined
    at getDefaultFetch (~/work/dfinity/node_test_2/node_modules/@dfinity/agent/lib/cjs/agent/http/index.js:57:28)

We can again fix this by patching the index.js file, adding the following line after use strict:

global.fetch = require("node-fetch");

And finally, after all this we get the call to work:

~/work/dfinity/node_test_2$ node app.js 
Hello, TEST!

Awesome!

Now, the question is, was I doing something wrong, or are there better ways of calling canisters from node.js? My eventual goal would be to run this on a service like heroku, so it would be great if this can get patched at source. (and of course someone should check if those hacks affect other areas of code)

21 Likes

Excellent writeup! You’re right - node.js workflows are underdocumented, and I’ll add it to my list.

At a cursory reading, you’re not doing anything wrong - we would expect most devs to do the same sort of TextEncoder and fetch setup. I think it can be easier with how to manage the canister ID, but generally you’re doing things right

2 Likes

You are a scholar and a gentleman. Thanks for the write-up!

wicked!

Thanks for this … got me over a hurdle … and its working in my local …

I can’t seem to find where the “production” host would point to.
i.e.
if "host: “http://localhost:8000” for the HttpAgent in dev environment, what address do I put in for a canister on the actual IC? …
I looked but could not find easily, and tried a few but cannot get it to work …
Thoughts/input?

Much thanks

1 Like

I haven’t tried it myself yet, but in the example provided here it’s:

<input type="text" id="hostUrl" value="https://boundary.ic0.app/" />

[...]

const actor = Actor.createActor(idlFactory, {
    agent: new HttpAgent({
      host: hostUrlEl.value,
      identity,
    }),
    canisterId,
  });

Also, I think that not passing any hosturl is defaulting to the public url as well. In the template created with dfx new the declarations .js file seems to work like this:

export const createActor = (canisterId, options) => {
  const agent = new HttpAgent({ ...options?.agentOptions });
 
[...] 
 
  // Creates an actor with using the candid interface and the HttpAgent
  return Actor.createActor(idlFactory, {
    agent,
    canisterId,
    ...options?.actorOptions,
  });
};

 export const test_assets = createActor(canisterId);

I’d try it first without a hosturl, and see if it goes to the public one.

double wicked! it works with the boundary address …

good looking out on the pointer to the example … I should have found that :slight_smile:

I had tried no host which errored … so I chased in the agent lib and the HttpAgent requires it with no reference to what is the production host …

Anyway it works … currently from command line, going to get it into a service now …

THANK YOU,
-M

1 Like

Did you ever get this working with an identity? How do you load in your local dfx identity into an identity?

I think you’ll get an answer in this thread Problems with cross referencing principals

I never tried that in JS, but yeah you can do that in rust. You can use your dfx identity .pem file to issue calls programmatically. Let me know if you need it now, otherwise I’ll make a post about it sometime next week. I want to write a longer thing about my experience in the hackathon, and will hopefully have some rust code that interacts with canisters. Just need to clean it up a bit.

1 Like

Hi everyone,
Have you managed to authenticate your calls while using node.js ? Is there a way to set the httpAgent identity with a .pem file ?
I want to run my script as the canister administrator.

Not in node, but I did that in rust if it helps. You can import your .pem file from the dfx identity and call canisters with that. Let me know if that works for you and I’ll look for the code.

It’s okay. I’ll write a bash script using the dfx cli. Thanks !

Y’all need to start following me on GitHub.

5 Likes

hey sir,
I have a question
Can I move my internet identity (on phone) to the laptop without yubikey and where I can see my seed phrase on phone?

This worked beautifully for me.

Since I’m on node v14, I didn’t need to set TextEncoder on the global object since it’s already set by default.

Also, you don’t have to patch node_modules/@dfinity/agent/lib/cjs/agent/http/index.js.

Instead, you can set fetch on the global object in your own module provided you set it before importing @dfinity/agent.

1 Like

Can you describe an easier way to manage the canister ID?

Right now, I import it from the dfx generated files:

import _SERVICE, {
  // @ts-ignore idlFactory is in service.js
  idlFactory as service_idl,
  // @ts-ignore canisterId is in service.js
  canisterId as service_id,
} from '../../../.dfx/local/canisters/service/service';

The ../../.. is not pretty, and TypeScript doesn’t know about idlFactory and canisterId since it’s exported in service.js and not service.d.ts.

FYI I’m still on dfx v0.7.1… not sure if this is an instance where upgrading dfx would make this more ergonomic.

At the end of the day, I’m managing it using environment variables and naming conventions, and having the bundler handle figuring it out from the canister_ids.json file.

There are plenty of other valid ways to manage it, but the design challenges I’m trying to manage are:

  • During development, the app should use the local canister id for canisters in the same project
  • When I specify the network, the frontend should point should automatically point to the canister IDs from that network
  • No code should need to be manually changed when deploying to those different targets.
  • Dfx should be involved as little as possible, and the implementation should stick to JavaScript patterns and conventions
3 Likes

Here’s the new Webpack config that will be coming with dfx 0.8.4 - I made the setup simpler and now it maps all the canisters dynamically, instead of requiring you to add them to the EnvironmentPlugin manually

1 Like

You won a follower :rofl:

1 Like

I hope it will be the last question :slight_smile: , is it normal that i don’t have the same Principal when using the CLI and when i’m using your script with my “~/.config/dfx/identity/default/identity.pem” file ?
I need the script to impersonate the “canister deployer”