Using dfx on NixOS?

Is anyone here working with dfx on NixOS?

Until dfx is open sourced we have to live with the binary that’s shipped in https://sdk.dfinity.org/downloads/dfx/0.8.1/x86_64-linux/dfx-0.8.1.tar.gz. The problem is that it’s not a static binary, and has /lib64/ld-linux-x86-64.so.2 hard-coded as the ELF interpreter.

I can create a nix derivation that fixes that for dfx, so I can run dfx. But dfx has this odd habit of unpacking further such binaries to ~/.cache/dfinity/versions/0.8.1/, and that happens at runtime, so I cannot patch it.

I created a nix derivation that, at build time, extracts these binaries, fixes their ELF interpreter, and then uses DFX_CONFIG_ROOT to make dfx use these, but that environment variable doesn’t just affect where dfx looks for the path, but also where it stores the identity, so letting that point to /nix/store doesn’t work well either. Too bad there is no way to set just the cache directory.

I guess waiting for dfx to be open source will be easiest.

@jwiegley, how do you do that?

Now that the sdk repo is open source I had another look, but the whole way how dfx handles assets (tar-ballling them, then embedding the tarball in the code, and at runtime extracting the members) is quite … opaque.

Am I really the only on in the intersection of people who may want to use dfx and who may want to use nix?

2 Likes

I created a repository with a rudimentary work-around, which could also be a place for a better solution:

2 Likes

I’m pretty confused about the current state of things here.

I thought @ericswanson fixed this in test: make sure shipped binaries don't reference /nix/store by ericswanson-dfinity · Pull Request #1868 · dfinity/sdk · GitHub (released in 0.8.3 according to the release notes)

However, I spent quite a while trying to get dfx working on Linux for CI via GitHub Actions and haven’t had much success.

My attempts can be seen on this branch although the CI runs are in another repo. GitHub - paulyoung/nixpkgs-dfinity-sdk at patchelf

What I can recollect is:

  • On 0.8.4, I was getting the “no such file or directory error” when trying to run ./dfx cache install, which was supposedly already fixed.
  • Patching the dfx binary with patchelf in the same way that the SDK does it allowed me to get past that but I ran into the same error for the binaries installed in the cache directory.
  • Patching the other binaries in the same way that the SDK does it only got me so far; I kept getting the same error for icx-proxy
  • On 0.9.2 I ran into some similar symptoms but it seemed like some of the binaries were now statically linked and patching wouldn’t apply so I wasn’t sure what to do.

I can probably do a better job of recreating all of this and investigating using ldd by getting a NixOS VM up and running again.

I don’t understand why patching would be necessary when it’s apparently already been done.

I also noticed the following, so maybe these are known issues?

To clarify, the macOS build works fine and I can run tests against canisters and a replica with ic-repl.

Dfinity patches the released binaries to replace the /nix path to the linker (which would not work elsewhere) with /lib64/ld-linux-x86-64.so.2 (which works elsewhere, but not on NixOS). So to use them on NixOS, you have to patch that back.

That’s easy enough for dfx itself, using some nix hook (patchElf or so), but tricky for the binaries that are embedded into dfx.

I thought I was taking all of that into account.

Hmm, ok, this extracts and patches these binaries, and puts them in the path. But wouldn’t dfx build or dfx start still extract and then call the unpatched binaries? What’s telling dfx to use these?

(I found no nice way of letting dfx use them, e.g. via an environment variable or such, and stopped short of patching dfx to allow that. Then it should work nicely, I believe.)

I’m pretty sure I know what to do. Will try and get things working this weekend.

@nomeata I believe I’ve fixed this with Patch binaries and add CI by paulyoung · Pull Request #3 · paulyoung/nixpkgs-dfinity-sdk · GitHub

Actually, no. If you try to do dfx deploy (for example) it fails because the config directory is now set to a path in the nix store.

I think what we need is a separate environment variable for the cache directory, i.e. DFX_CACHE_ROOT.

1 Like

I’ve created an issue to track this and plan to propose a change to dfx to address it, unless some enthusiastic Nix user beats me to it.

Blocked by dfx development depending on a private repo :frowning_face:

I created an issue instead.

Beh, yes, that’s also what I ran into. Patching dfx to be more friendly is probably the best way. Maybe even a envvar for each binary to override what to use? Might be useful to others as well.

Here’s a github workflow that runs dfx both on linux and macos: sdk/e2e.yml at master · dfinity/sdk · GitHub

Starting with the next dfx release, we will start releasing dfx as built from a github workflow (sdk/publish.yml at master · dfinity/sdk · GitHub), rather than built using nix.

This is looking promising!