Request for Comments: API Changes for Public & Private Neurons

Now that Public & Private Neurons has been approved by motion proposal, I wanted to share more details with developers how the API would change, and request feedback. This will cover not only what the API would eventually look like, but also how we would transition from the current API to the eventual API, paying special attention to breaking changes and migration strategy.

Breaking Changes

The breaking changes would affect apps that read neurons that do not “belong” to the caller (i.e. the caller is neither a controller, nor hotkey of the neuron).

How the behavior will be different in the eventual API: when a) you call call a method whose response contains NeuronInfo (e.g. get_neuron_info), b) the neuron does not belong to you, and c) the neuron is private, then, a couple of fields will eventually be “redacted”. Specifically,

  • recent_ballots: This field is of type vec BallotInfo. Here, redacted means that the field will have 0 elements.
  • joined_community_fund_timestmap_seconds: This field is of type opt nat64. Here, redacted means that the field will have null as its value.

To interpret these values, apps will need to make use of a new field: visibility. It will be of type opt int32. Possible values in this field, and what they mean:

  • opt 1: Private. In this case, if recent_ballots has 0 elements, it does not (necessarily) mean that the neuron did not vote recently. Rather, the caller cannot tell whether the neuron voted recently. Similarly, if joined_community_fund_timestamp_seconds is null, it does not (necessarily) mean that the neuron is not a member of the Neurons’ Fund. Rather, the caller cannot tell whether the neuron is in the NF.
  • opt 2: Public. In this case, the aforementioned fields would not be redacted.
  • null: Visibility enforcement has not been released yet. See the next section “How to Migrate”.

Structurally, the interface changes would look like this:

type NeuronInfo {
  // Existing code elided...

  visibility : opt int32;
}

Notice that even apps that do not migrate would still be able to deserialize responses, because the only structural change is the addition of visibility (which is of type opt int32). They would simply misinterpret the redacted fields once visibility enforcement has been released.

How to Migrate

To give apps an ample migration window, the API changes would be released in multiple steps. For breaking changes, two releases would be of particular interest:

  1. The visibility field would be added to NeuronInfo. This would have one of the values described above, but at this point, there is no redaction, no “enforcement”. At this point, the governance canister would declare this new visibility field, and apps would be able to start reading it, and using it to get ready for the second step.
  2. Fields are redacted. At this point, we say that visibility is “enforced”.

Thus, the migration window lies between these two points. During this time, apps would need to add code that looks like this pseudocode:

if recent_ballots.is_empty() {
  // New: use visibility to deduce why we see recent_ballots is empty:
  let is_face_value =
    visibility == null || // No enforcement yet.
    visibility == opt 2;  // Neuron is public.
  if is_face_value {
    print("The neuron has not voted recently.");
  } else {
    assert visibility == opt 1; // Neuron is private.
    print("Not sure if the neuron voted recently, because it is private.");
  }

} else {
  // Old behavior.
  print("Here are the neuron's recent ballots:");
  for ballot in recent_ballots {
    print_ballot(ballot);
  }
}

Notice that during the migration window, unmigrated apps would still behave the same.

Notice that migrated apps will continue to work correctly after the migration window without further changes.

Non-Breaking Changes

This section describes new functionality that apps can start taking advantage of (if they want to).

Reading and Writing Visibility

Earlier, we mentioned that NeuronInfo would gain a new visibility field. The Neuron type would also gain this field. As explained in the previous section, initially, visibility would not be “enforced”.

In addition to being able to read a neuron’s visibility, users will also need the ability to write the visibility of neurons that belong to them. To do that, there will be an enhanced version of the manage_neuron method. Specifically, the following will be added to the Operation type:

type Operation = variant {
  // Existing code elided...

  // New.
  SetVisibility : SetVisibility;
}

// New.
type SetVisibility = record {
  visibility : opt int32;
}

(The visibility field within SetVisibility is analogous to the visibility field in Neuron, except that null will be considered an invalid value, and the manage_neuron call will result in an Error response.)

Reading Public Neurons

Currently, one has to have special privileges to read a full neuron, but once we have public neurons, special privileges will not be necessary to read a full neuron if it is public. To accommodate this, the list_neurons method will be changed as follows:

  • ListNeurons.include_public_neurons_in_full_neurons
    • This is a new field that controls the behavior of the list_neurons method.
    • type: opt bool
  • ListNeuronsResponse.full_neurons
    • Existing field.
    • Previously, this only included neurons that belong to the caller.
    • However, going forward, callers can use include_public_neurons_in_full_neurons to specify that they want public neurons to also be included in this field.

Structurally, the changes would look like this:

type ListNeurons = {
  // Existing code elided...

  // New.
  include_public_neurons_in_full_neurons : opt bool;
}

Known Neurons Are Always Public

This will happen automatically. If a known neuron tries to become private, they will get an Error response.

References

This feature was discussed in an earlier thread.

6 Likes

Thanks for sharing this.

Did I understand correctly that once this is released, none of the neurons will become private automatically without any specific action from the user?

1 Like

I was wondering if there is an estimated release date for the Neuron Index? Any information would be greatly appreciated!

1 Like

:grin:

No. Per the motion proposal, they will be private.

We will do our best to encourage neurons to set themselves to public. Our ideas around this mainly involve nudging in the NNS Dapp.

The best answer I can give to this right now is “in the not too distant future”. FWIW, the reason we are doing this is to prepare the way for neuron index.

1 Like

Ok, found it in motion proposal text:

Each neuron can choose to be private or public and change this at any point in time. The known neurons are public while all other neurons are set to private by default.

TL;DR: API consumers can begin migrating right now.

(Actually, this :point_up: was true earlier, but I didn’t update this thread as soon as possible.)

Most of the backend API changes have been released.

Next Steps

  • The main thing left is to enable redaction of fields in NeuronInfo for private neurons.

  • Another small thing is left to do is enable SetVisibility (ManageNeuron) proposals.

We do not have very specific plans when these things would happen. Ideally, sooner is better, but since enabling redaction could be disruptive to some API consumers, we want to make sure everyone has a reasonable amount of time to migrate. Please, reply if you have concerns about when these changes are released.

1 Like

Updates:

  1. We already enabled SetVisbility proposals.

  2. We think it should be ok to enable private neuron enforcement next week, and that’s when we plan to do it.

1 Like

Sorry for no recent updates. We discovered a blocking issue, but that is now in the process of being cleared. It is very likely that we will make a proposal in 7 days that will enable private enforcement.

1 Like

Private enforcement is in the upgrade proposal that was created on Friday.

The proposal that I mentioned in my last post was executed successfully. Therefore, I declare this feature “fully launched”.

There is some code clean up work that remains to be done, but none of it will result in any behavior changes.

Thank you for your continued support of the Internet Computer.

2 Likes

Great news! Thanks @daniel-wong. Is there anywhere where everything about public neurons is made available yet, or is this yet to be introduced on the neurons dashboard (or elsewhere)?

You can always call the NNS Governance canister directly.

We plan to add newly exposed neuron fields (for both public and private neurons) to the ICP Dashboard, but this won’t be added until the neuron-related pages on the dashboard are redesigned.

1 Like

Hey @daniel-wong ,
Since the private/public neurons were just part of Proposal #130832, we’re looking for the neuron index and need some guidance on accessing it. Could you please point us to the canister/method for querying the full neuron list?
Thanks.

1 Like

Thanks @Dylan :slightly_smiling_face: Assuming I’d like to see the principal for a specific neuron (for example, in order to validate claims of decentralisation made by a community-controlled neuron), what method do I need to call on the NNS governance canister.

I just executed get_neuron_info, but this doesn’t appear to provide the principal for the neuron (regardless of visibility = opt (2 : int32); )

Is there another method I need to call to get this?

Good question. Based on testing I just did, it seems like you need to call list_neurons with the neuron ID and set include_public_neurons_in_full_neurons to true.

cc: @daniel-wong (in case there is another way to get that data)

2 Likes

Sorry, I am not sure when that is coming. I think @lara might have a clearer idea.

The neuron index is not implemented yet as there is some more work to be done before we can work on it. It would probably be Q1 2024 when we’ll be able to address this.