Dfx deploy asset upload ignores hidden directories

First off, I’m using https://www.snowpack.dev/ and not Webpack, so I am not bundling everything into one file.

A problem I have run into is that dfx deploy is ignoring hidden directories inside of asset canister upload directories. It will not upload them to the canister. This is causing problems for me when using Snowpack, messing up my import aliases, as I have to alias into the .dfx directory, which is technically hidden. Snowpack takes this directory and potentially builds it and puts it into my asset canister upload directory.

I think it’s unexpected that dfx deploy would ignore hidden directories. For example, git does not ignore hidden directories unless you explicitly ignore those directories. I would expect all directories and files in the asset canister upload directory to be uploaded to the directory, and perhaps to have some sort of ignore configuration for everything else.

1 Like

We’ve got a proposal to add a command to dfx that will generate the interface files and output them into your source code specifically for a nicer setup with bundlers like snowpack.

For now, you can add didc commands to your package.json and get creative with it. Check out the implementation in GitHub - dfinity/internet-identity: Internet Identity, the Single Sign On service for the Internet Computer where we needed to do the same

1 Like

Very nice, I can see what didc is doing. But where can I find and install didc?

I found this page: Neuron/ICP Instructions for Seed Participants :: Internet Computer

That has some information about didc, and I found the repo here: GitHub - dfinity/candid: Candid Library for the Internet Computer

I was able to get it installed

I have been able to get Snowpack to work and I am able to communicate with my canister! Thanks!

I still think hidden files and directories perhaps should not be excluded by default.

Yeah, I can see that going either way. The consequences of accidentally publishing something that was deliberately made secret seem to be a worse outcome to me, personally

1 Like

Another alternative is a way to override that…something like .gitignore, with git nearly everything is included by default and then you specifically exclude. Even if dfx excluded some things by default and allowed the user to explicitly include, that would be nice. But right now there is not way to control that

Has this issue been resolved? I’m running into a similar problem. I use SvelteKit to build a website. I have two hidden files .well-known/ic-names and .ic-assets.json in the output directory that’s defined as the source in dfx.json. Every other file in the source directory gets picked up by dfx deploy except these two hidden files (which are needed to configure custom domain name for my site). It seems that dfx doesn’t process hidden files? Is there a way to impose that?

This issue surfaced when @rikonor was helping me figure out how to set up the custom domain.

Did you whitelist .well-known in .ic-assets.json?

e.g. https://github.com/buildwithjuno/juno/blob/main/src/frontend/static/.ic-assets.json

I did whitelist both files in the static folder of SvelteKit. Both files can persist in the final output directory that SvelteKit renders via npm run build. However, dfx deploy somehow didn’t pick up these two files and didn’t upload them to the canister (every other file got picked up just fine).

Would you mind sharing your .ic-assets.json file? I suspect the error is in there

~/github-zire/hyxyz 21:35 zire@arrakis $ lsl
total 64
755 drwxr-xr-x   16 herbert.yang  staff    512 Feb 21 09:38 .
755 drwxr-xr-x   19 herbert.yang  staff    608 Feb 19 19:52 ..
644 -rw-r--r--@   1 herbert.yang  staff  10244 Feb 19 19:51 .DS_Store
755 drwxr-xr-x    8 herbert.yang  staff    256 Feb 19 16:23 .dfx
755 drwxr-xr-x   15 herbert.yang  staff    480 Feb 20 16:35 .git
644 -rw-r--r--    1 herbert.yang  staff    165 May  5  2022 .gitignore
644 -rw-r--r--    1 herbert.yang  staff   3538 Oct  9 14:34 README.md
644 -rw-r--r--    1 herbert.yang  staff     60 May  5  2022 canister_ids.json
644 -rw-r--r--    1 herbert.yang  staff    323 Oct  9 10:54 dfx.json
755 drwxr-xr-x   21 herbert.yang  staff    672 Feb 21 09:43 herbertyang
755 drwxr-xr-x   21 herbert.yang  staff    672 Oct 13 21:39 herbertyang-tmp
755 drwxr-xr-x@ 111 herbert.yang  staff   3552 Dec 27  2021 pelican_convert
755 drwxr-xr-x@ 323 herbert.yang  staff  10336 Oct 11 01:03 pelican_images
755 drwxr-xr-x@  12 herbert.yang  staff    384 Oct 11 00:51 pelican_posts
755 drwxr-xr-x    3 herbert.yang  staff     96 Oct 14 01:46 sveltekit-blog-starter
644 -rw-r--r--    1 herbert.yang  staff    155 Feb 20 16:04 tmp
~/github-zire/hyxyz 21:35 zire@arrakis $ cat herbertyang/static/.ic-assets.json
[
    {
        "match": ".well-known",
        "ignore": false
    }
]
~/github-zire/hyxyz 21:35 zire@arrakis $ cat herbertyang/static/.well-known/ic-domains
herbertyang.xyz
~/github-zire/hyxyz 21:35 zire@arrakis $ cat dfx.json
{
  "version": 1,
  "dfx": "0.9.2",
  "canisters": {
    "hyxyz": {
      "type": "assets",
      "source": ["herbertyang/build"]
    }
  },
  "defaults": {
    "build": {
      "packtool": "",
      "args": ""
    }
  },
  "networks": {
    "local": {
      "bind": "127.0.0.1:8000",
      "type": "ephemeral"
    }
  }
}

Does any asset in herbertyang/static/ get uploaded? I suspect you simply haven’t told dfx to include that folder. In that case you’d have to change dfx.json’s definition of the canister sources like this:

  "canisters": {
    "hyxyz": {
      "type": "assets",
      "source": [
           "herbertyang/build",
           "herbertyang/static"
      ]
    }
  },

Then dfx should also look in the static folder for things it’s supposed to upload

I just tried that, by adding herbertyang/static as another path in “source” in dfx.json. After running npm run build and dfx deploy --network=ic --no-wallet, the result is still the same though. It says, “domain is missing from canister xxx”.

* We are completely uploaded and fine
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 500
< server: nginx/1.21.3
< date: Wed, 22 Feb 2023 00:27:20 GMT
< content-length: 81
< access-control-allow-origin: *
< access-control-allow-methods: HEAD, POST
< access-control-allow-credentials: true
< access-control-allow-headers: DNT,User-Agent,X-Requested-With,If-None-Match,If-Modified-Since,Cache-Control,Content-Type,Range,Cookie
< access-control-expose-headers: Accept-Ranges,Content-Length,Content-Range
< access-control-max-age: 600
<
* Connection #0 to host ic0.app left intact
domain is missing from canister hbc6w-gqaaa-aaaag-aagdq-cai list of known domains

@rikonor helped me investigate this error message yesterday and confirmed that it was because the two hidden files, .well-known/ic-domains and .ic-assets.json were not uploaded to the canister. From dfx’s execution messages, I can also tell that every other file in herbertyang/build and herbertyang/static was uploaded, except for these two.

Just thinking out at loud, are you on Windows?

If yes, does the .ic-assets.json whitelist for .well-known works on Windows?

@peterparker I’m on a Mac.

A pity. Don’t have more idea. Maybe the best is to share your repo with the team so that they can try to reproduce?

Found the issue. My dfx version was too told (duh … :roll_eyes:). I upgraded dfx from the original 0.9.2 to 0.12.1 => fixed the (new) issue with the canister principal (not a problem in the older version of dfx) => removed herbertyang/static from source in dfx.json (new dfx screams for sync error … not a problem in the older dfx) => the rest was a breeze.

My site is now live at https://herbertyang.xyz . No more funky https://hbc6w-gqaaa-aaaag-aagdq-cai.raw.ic0.app .

Thank you @samshi @rikonor @Severin @peterparker for helping me narrow down the problems, isolated other issues, and identified the fix! Much appreciated.

4 Likes

Happy to hear you resolved it and looks cool :+1:.

Also a valudable feedback for my personal project. With Juno I took another approach. Everything in static is deployed unless developer ignores explicitely resources. In addition, the ic-domain is automatically generated. So both issues you faced would have not find place :wink: .

2 Likes

Before, I encountered the same error as Herbert: “domain is missing from canister kqlfj-siaaa-aaaag-aaawq-cai list of known domains.” However, I found that I just needed to include the .ic-assets.json and .well-known files in the build. I am now able to download the file at https://kqlfj-siaaa-aaaag-aaawq-cai.ic0.app/.well-known/ic-domains. However, I am now facing an “unexpected error.” I am not sure if anyone else has encountered this issue. Using google domains.