External contributions to the SDK are blocked by private dependencies

Developers Command Line Tools
1

The dfinity/sdk repo depends on a private dfinity-lab/common repo.

This means that anyone without access to that repo who would like to contribute to the SDK immediately runs into the issue below when following the contributing guidelines.

2 Likes
2

Sharing here for some additional visibility since the issue is 6 months old and there doesn’t seem to be any sign of changing things yet.

2 Likes
3

Yes, would be good to have this fixed. Given that the ic repo doesn’t use common anymore, maybe simply copy the relevant files to sdk to make it self-contained? Surely some engineer at DFINITY can do that more quickly than going through a full open sourcing exercise for a repo that probably nobody really cares much about any more.

1 Like
4

The contributing guidelines are out of date. It’s now possible to build dfx with cargo build, no nix or common needed.

1 Like
5

But it’s nice to use nix :slight_smile:

1 Like
6

Having just submitted feat: Introduce DFX_CACHE_ROOT environment variable by paulyoung · Pull Request #2112 · dfinity/sdk · GitHub I can say that it would have been great to be able to use the Nix derivation provided for running the end-to-end tests locally.

1 Like
7

Guidelines have been updated. No nix mentioned anymore. Have fun!

1 Like
8

@paulyoung, guess they really really don’t want us to use the nix file there. We may have to buidRustPackage it externally, essentially recreating the build setup…

9

Pretty much - you’re welcome to use Nix for your own setup, but we’re committed to removing it as a requirement/expectation in projects that accept community contributions

10

Oh, this isn’t about requiring external contributors to use nix, merely allowing it at least as long there is a nix setup in that repo. (If the sdk repo didn’t have a working nix setup we wouldn’t be asking for one, but it seems it has one that we can’t use for unsatisfying reasons.)

Of course if the nix setup in the sdk repo is stale and doesn’t work anymore anyways, then that’d be a different thing. But then it probably shouldn’t be there anymore… :slight_smile:

2 Likes
11

Being able to build from source in a way that’s guaranteed to be reproducible would help in situations like the one I’m in now where a PR has been merged but the changes haven’t been released yet.

If the derivations in the SDK repo are being phased out I’d like to take what’s there now and get them working without the private dependencies.

I’ve seen the benefit of using Nix for the past 6 years and would like to keep using it for my own projects.

@ericswanson @Severin do you have any advice on what it would take to replace what’s missing from dfinity-lab/common? I attempted to do this recently by stubbing things out and following errors but didn’t get very far.

@nmattia, would you or someone else familiar with Nix be willing to help?

12

if the nix setup in the sdk repo is stale and doesn’t work anymore anyways

This is closer to the truth. It works, but there’s come maintenance burden associated with making dfx build both with nix and with cargo build outside of nix.

Some of those issues might be more about building under hydra or other underlying problems rather than about building in nix itself.

Examples:

13

@nmattia, would you or someone else familiar with Nix be willing to help?

I’m not sure this would be such a good idea, here are some observations I made when hacking on the repo last week:

  • I was able to get a working shell using this: nix-shell -p cargo -p darwin.apple_sdk.frameworks.Foundation -p pkg-config -p openssl -p libiconv (macOS)
  • All .nix files should probably be removed if they’re not used, right now they’re just confusing, and
  • I was extremely surprised to have a build.rs actually downloading things; I don’t expect the build process to download things or generally need access to the internet.
14

Agreed, I’ll discuss this with @ericswanson and vote for this.

I speculate this is the alternative to internal nix dependencies.

15

This is exactly the issue I ran into which unfortunately means I’ll probably abandon the idea of building from source.

16

If having build.rs download things is a problem (which I have run into on certain environments, for example the musl build action in a GitHub workflow), you can:

  • run ./scripts/prepare-dfx-assets.sh <some path>
  • set the DFX_ASSETS environment variable to that path

Then cargo build via build.rs will use the assets that are already prepared, rather than calling that script itself.

The reason for build.rs to do this in the first place is so that cargo build works after cloning the repo, or in IDEs, or anywhere else that you’d expect cargo build to work, without further setup, and also updates those assets after cargo clean or git fetch or other actions that change the inputs for those assets.

2 Likes
17

The prior state that you could only build with cargo build if you’d also installed nix, and used that to build (download) these assets, and set DFX_ASSETS to point to the the resulting output directory. We wanted to make it so you could build by cloning the repo and running cargo build , which seems like a reasonable expectation for a rust project.

18

For the record, I’ve described a potential solution on the GitHub tracker.