Getting data into an IC "database"

I’m using a canister to act as a “database” for my podcast-serving app. While building, I’ve been running a local instance (dfx start) and have been manually adding some test data via the candid UI. For larger test data sets, I have a script which generates a candid string and uses the command line interface to get the data into the canister.

Now I’m ready to deploy the canister and make it available to the world. Once deployed, there isn’t a candid interface for me to use to insert data (and that’s not scalable anyway…) so that’s not an option. Using the CLI won’t work either, because I’d need to be running the command from the same machine that’s running the IC network. This works locally, but I can’t run all my data-processing scripts on the IC, so I’m not sure how to get data into a canister running anywhere other than my local machine.

For reference, I have scripts that collect videos from various sources (with permission from the owners) and add them to podcast feeds. These scripts need to be able to modify the state in my canister as it runs on the network.

I could make a frontend that allows users to add data manually, but I’m looking for a way to do it automatically using scripts written in Python or Dart, for instance.

The only way I’ve thought to do this is to expose the add functions through http access using @nomeata’s ic-http-bridge. Is this the only way to go about doing this? Or is there a better way to allow adding data via scripts, either my own or other “admin” users?

What about creating a function on your canister that accepts a string that you create out of your data and parses it into the canister-data-structure?

That’s what I’ve been doing, and using a command like dfx canister call myCanister addData '("The","Data")'. But that only works for canisters running locally on the machine making the call. Otherwise, how would it know where to find the canister?

dfx deploy supports a --network ic flag to deploy canisters to the network, but there doesn’t seem to be support for this flag in the dfx call command. What’s the syntax for calling a function deployed on the ic network?

No way they missed that lol:
dfx canister --network ic call <canister_id> greet levi

<canister_id> is the one you get when you deploy to the non-local ic btw, different canister ids for each network(local and IC)

for some reason now that im testing it, that command only works from within that specific canisters project directory, im not sure why if its deployed on the IC should be able to call functions from anywhere as long as you have the canisters IC ID? you can still change the directory from a python script or something for your case though the command works.

As @levifeldman says, if you pass --network ic to canister, then it works with call just fine.

It is an oddity that you can’t use dfx at all without a project repository… even if you call other canisters (by their canister id), it still needs the dfx.json to know what the “ic” network means.

You can play around with the icx tool in agent-rs/icx at next · dfinity/agent-rs · GitHub, which is a more low-level tool than dfx, i.e. only focuses on interacting with the IC, but not building canisters etc.

2 Likes

@levifeldman Ahh, ok. That was my dumb mistake.

This works:

dfx canister --network ic call <canister_id> greet levi

This doesn’t:

dfx canister call <canister_id> greet levi --network ic

The help text printed for dfx deploy --help mentions that --network is an option, but that option is missing from dfx canister call --help. I didn’t notice it on dfx canister --help and presumably it works on all subcommands of dfx canister.

I also noticed that it works without specifying the canister id; you can just specify the name of the canister. When I ran dfx deploy --network ic, it created a new canister_ids.json at the project root (i.e. not under the .dfx folder) with the cid listed under ic for each of the canister names. It must use this cid when you specify --network ic for commands like dfx canister call.

3 Likes

@mymikemiller I feel you, it is not so clear.

I agree this could be clearer. (@stanley.jones)

1 Like

I have updated ic.nomeata.de to a 45s timeout. Enjoy!

1 Like