Is it possible to Dynamically install code to FRONTEND canisters?

Did a bit more digging. Each day it all makes a bit more sense. Now I’m getting ready to make the helper methods that will assemble the operations that will ultimately be put into the commit_batch method as an argument. I put together a game plan for programmatically updating the assets of front end canisters. I’d like to run it by you all here before I write all the code. I’ll be using the list({}) method (within the asset canister) and type BatchOperationKind below, I’ve listed the function signature and the type definition:

list ({}) -> ([{
                key: Key;
                content_type: Text;
                encodings: [{
                content_encoding: Text;
                sha256: ? Blob; // sha256 of entire asset encoding, calculated by dfx and passed in SetAssetContentArguments
                length: Nat; // Size of this encoding's blob. Calculated when uploading assets.
                modified: Time;
            }];
        }])

The BatchOperationKind is defined like so:

type BatchOperationKind = variant {
  CreateAsset: CreateAssetArguments;
  SetAssetContent: SetAssetContentArguments;

  UnsetAssetContent: UnsetAssetContentArguments;
  DeleteAsset: DeleteAssetArguments;

  Clear: ClearArguments;
};
type CreateAssetArguments = record {
  key: Key;
  content_type: text;
  max_age: opt nat64;
  headers: opt vec HeaderField;
  enable_aliasing: opt bool;
  allow_raw_access: opt bool;
};
type SetAssetContentArguments = record {
  key: Key;
  content_encoding: text;
  chunk_ids: vec ChunkId;
  sha256: opt blob;
};
type UnsetAssetContentArguments = record {
  key: Key;
  content_encoding: text;
};
type DeleteAssetArguments = record {
  key: Key;
};
type ClearArguments = record {};

The plan is as follows:

1.) Maintain a master copy of the asset canister. this asset canister will be within my control and I will be updating it via the DFX SDK command.

2.) create a store canister. the store canister will also be a controller of the master copy asset canister, thus authorizing it to retrieve the assets data that the master copy holds. I’ll be using the list({}) method (within the master copy of the asset canister) to retrieve the assets data. Once the store canister retrieves the updated assets data from the master copy, it will then store that data within its stable memory. There, the updated assets data can be retrieved by any canister via a shared method.

3.) Create a manager canister for each of the asset canisters that are due to receive updates. The manager canisters will each be a controller of their own respective assets canister. The manager canister will be programmed to: first, pull the assets data from the store canister, then manipulate the data so that it conforms to the type BatchOperationKind, and finally commit those updates to their respective assets canisters via the commit_batch method that exists within the assets canisters.

is there any reason why this wouldn’t work? particularly with respect to step 3? is there any reason why i wouldn’t be able to use the data that I get from the list({}) method’s result in order to produce the type CreateAssetArguments or the type SetAssetContentArguments? my worry is that sha256: opt blob may be created such that the canister for which the sha256 hash was created is the only canister that is able to interpret/decrypt it, and as a result, using it to update the assets of another canister would cause issues. @peterparker, @Severin

1 Like

Sounds reasonable to me. I’d like to propose another functionality for your manager canister: It should be able to update the asset canister’s wasm as well, not just the content. Unless of course you have that functionality already through some other mechanism. We regularly release new asset canister versions and you may want to have the new functionality available as well.

You told me that you won’t have that much data, but I’d like to recommend you test your manager canister with substantially more data. E.g. like in the test added here. That way you can make sure you don’t have any mistakes with chunking larger assets in case they would ever appear in your app.

No need to worry. The sha256 is not used in the asset canister itself. It’s there for the service worker so that it can verify the content it received. And we can’t calculate the hash for every asset in the asset canister itself because we’d run into execution limits, therefore it’s supplied during the upload as part of the asset.

This raises another question. What does the list({}) return when one of the asset’s data is chunked? I assume the list({}) function would only return data for the first chunk of each asset and to get the rest of the data for each asset, you’d have to use the get_chunk() method. Is that a proper assumption?

What prevents the vector returned by the list({}) function from being larger than 2MB?

Side note: the argument for list should be (record {}), not ({}) in proper candid syntax in case you want to use dfx to call it manually

Anyways… I wasn’t sure, so I just tested it out. If you want to test yourself, follow theses steps:

dfx new hello
cd hello
dfx start --clean --background
dfx deploy
for a in $(seq 1 400); do dd if=/dev/urandom of=src/hello_frontend/assets/large-asset-$a.bin bs=2097152 count=1; done
dfx deploy

then you’re ready to call dfx canister call hello_frontend list '(record {})'

More note to self than responding to your question: Turns out list does not serve the asset’s content, only the sha and the length of the content. You can then fetch individual chunks with a call like dfx canister call hello_frontend get_chunk '(record {key = "/large-asset-2.bin"; content_encoding = "identity"; index = 1; })'. You can get key and encoding from the call to list, and the index you just count up from 0 until you received all bytes or until the next one traps with chunk index out of bounds.

If it has to be chunked it won’t tell you anything special. From this line I would assume that any asset with a length larger than 1_900_000 bytes will be chunked.

Nothing. list itself does not have chunking - this will be a problem at some point in the future. But since it does not serve the asset content itself, it’s unlikely that you would hit the limit - so far I haven’t heard of anyone hitting that problem. With some guesstimating and back-of-the-napkin math I’d expect any project with <6.500 assets to be safe, if you have exclusively assets that don’t get gzipped as well (non .txt, .js, .html according to this) then you can even go up to 13.000 and still be on the safe side.

2 Likes

@Severin i appreciate the help :pray:t5::pray:t5::pray:t5:. I’ll get to work

ok @Severin . I’m in the final stretch of getting my dapp to dynamically upgrade the asset canisters. I’m attempting to programmatically upgrade the wasm module of an assets canister. I’m getting the following error message when I do so:

Call was rejected:
  Request ID: 19106010cdeae865d8216386d484a14fbda3740af1a6f04f4e7db92683b1fb6b
  Reject code: 4
  Reject text: Unauthorized access. Caller is not the owner.

i noticed that the assets canister has a grant_permission method and a take_ownership method. I assume these will be necessary for resolving the error message I’m getting.

I used the grant_permission method to grant #ManagePermissions to the canister that will be delivering the upgrade to the asset canister’s wasm module. Then, from within the canister that will be delivering the upgrade to the asset canister, I attempted to call the take_ownership method of the asset canister, but I’m still getting the error message above. Is there something I’m missing?

This requires the caller to be a controller of the canister, which is also what the error message tries to say (but I think the wording is not that great). This refers to the list of principals you can see if you do dfx canister info <id>

No, these functions are for WASM-level (you can also call it ‘user space’ or ‘application space’) defined roles/permissions. They are not powerful enough to install WASM. For that you need the protocol-level control over the canister

1 Like

The canister that is attempting to install the wasm to the asset canister is indeed a controller of the asset canister. It still renders that error message though.

I don’t think that error comes from any DFINITY-produced code. I can’t find the string Caller is not the owner at all in any of our code bases. Reject code 4 usually means that the canister rejected the message, usually in a guard method. Is there a chance you are running into a guard somewhere in pre_upgrade, post_upgrade, or init?

You were correct. It was something on my end that needed to be debugged. My last question (i hope) is pertaining to the batch operation below:

public type CreateAssetArguments = {
        key: Key;
        content_type: Text;
        max_age: ? Nat64;
        headers: ? [HeaderField];
        enable_aliasing: ? Bool;
        allow_raw_access: ? Bool;
    };

what do I set the enable_aliasing variable to?

every other property, I was able to retrieve using the asset canister methods, but enable_aliasing appears to be the only property that I’m unable to find.

I’m not getting any errors, but the after updating the assets canister programmatically, the web page displays a blank background. I’m wondering if the enable_aliasing property has anything to do with that

I would simply always set enable_aliasing to true. I don’t see any case where it would break something, and I haven’t heard of anyone actually disabling it

For debugging, I’d have a look at your network requests if nothing shows up in the console. I like to use the ic-inspector extension to look at the actual requests to http_request. It’s very hard to say what’s going wrong from your description.

@Severin disregard this. I compared the requests from the non-working copy with those of a working copy. The issue has something to do with the some of the requests being missing. I’m noticing that all of the requests for assets that are .png file types are missing. Would you have any clue as to what could be causing requests of this type to not fire?

If the request isn’t firing at all then it sounds to me like a frontend issue. Do you have any idea what could stop the code from reaching the line where the request is made?

Since you mention streaming calls could it be that some of your assets are too large (>1900000 bytes IIRC) for a single message? Streaming support is mediocre at best and only works over raw.icp0.io for now

my /index.js asset is larger than 1900000 bytes. However, this request http_request_streaming_callback is executing with no problem in both the working example and in the non-working example. I ended up removing the .png files from the canister to reduce the size of the /index.js asset. Now the /index.js asset consists only one chunk, but that hasn’t resolved the issue.

the screenshot below is the error message that I’m getting in the console. Does this mean anything to you?

also, for the assets that are gzip’ed, do I need to unzip them before uploading them into the canister?

another suspicion i have is with the content_encodings. I suspect I may need to regenerated them as opposed to using the content encodings from the master copy.

@Severin , i was able to get it working successfully. The issue was that i wasn’t uploading all of the data chunks the asset canister.

1 Like

hey Jesse can you share uploading code example? i’m also working on this canister.

It was a whole process getting this stuff to work. I could upload the code, but I’m sure it make little sense. I’d be willing to jump on a call to explain it to you.