Seed neuron spawn failures - multiple neurons stuck in Spawning state

Developers
1

I have some seed neurons. I am trying to spawn the rewards into a separate neuron to access my ICP, but into issues.

First neuron:

Parent neuron IDs: 281525030204895507

Spawned neuron IDs: 358429794222158868

The transaction is successful, but I seems like the spawned neuron is in a corrupted state. I even tried to increase dissolve delay to 1 day, but can’t start dissolving.

Tried with a second seed neuron and the exact same issue.

Parent neuron: 685705572931231473

Spawned neuron: 9973304611228297795

It seems like a bug in the seed neurons. Can someone help? @SupportDev

Original error message on console that I saw: NNS error: There isn’t enough maturity to spawn a new neuron due to worst case maturity modulation.

More info: (on the second neuron that I tried):

$ dfx canister --network ic call rrkah-fqaaa-aaaaa-aaaaq-cai get_full_neuron ‘(9973304611228297795:nat64)’

(

variant {

Ok = record {

  id = opt record { id = 9_973_304_611_228_297_795 : nat64 };

  staked_maturity_e8s_equivalent = null;

  controller = opt principal "w6sf3-igbyf-gokg5-mp3hm-r3rxj-ysfkz-hogs6-i64hk-wtszh-yrgse-lae";

  recent_ballots = vec {};

  voting_power_refreshed_timestamp_seconds = opt (1_762_677_301 : nat64);

  kyc_verified = true;

  potential_voting_power = opt (0 : nat64);

  neuron_type = null;

  not_for_profit = false;

  maturity_e8s_equivalent = 33_454_665_363 : nat64;

  deciding_voting_power = opt (0 : nat64);

  cached_neuron_stake_e8s = 0 : nat64;

  created_timestamp_seconds = 1_762_677_301 : nat64;

  auto_stake_maturity = null;

  aging_since_timestamp_seconds = 18_446_744_073_709_551_615 : nat64;

  hot_keys = vec {};

  account = blob "\\c7\\9d\\e7\\c3\\ce\\fc\\56\\fe\\aa\\44\\d7\\dc\\82\\a9\\c9\\3c\\03\\d6\\cb\\e4\\c9\\6b\\19\\0c\\34\\b8\\aa\\9e\\56\\67\\88\\09";

  joined_community_fund_timestamp_seconds = null;

  maturity_disbursements_in_progress = opt vec {};

  dissolve_state = opt variant {

    WhenDissolvedTimestampSeconds = 1_763_282_101 : nat64

  };

  followees = vec {};

  neuron_fees_e8s = 0 : nat64;

  visibility = opt (1 : int32);

  transfer = null;

  known_neuron_data = null;

  spawn_at_timestamp_seconds = opt (1_763_282_101 : nat64);

}

},

)

So seems like maturity_e8s_equivalent is not 0 for some reason, and expecting cached_neuron_stake_e8sto be around 300 ICP.

2

Rewards are no longer spawned into neurons. The maturity is disbursed directly to an ICP wallet address rather than into a neuron. This behaviour was altered several months back, so may explain the issues you’re having.

3

Oh damn. I am using quill and dfx on the CLI and neither tool stopped me from spawning a new neuron. It even spawned the neuron with a new id.

I am running this via a .pem file. Any idea how I could access my ICP -

  • Access ICP from the 2 neurons that I already spawned for rewards.
  • Access ICP rewards from the other neurons that have been staking for around 4 years.
4

It doesn’t look like your spawned neuron has any ICP in it. I suspect the spawn suceeded at the neuron creation stage, but failed to move any maturity into it (just a guess as I’ve not looked at the code).

You should check if any of your maturity has actually moved (it probably hasn’t). @jasonzhu should know more.

5

Thanks @Lorimer but it seems like maturity was transferred / destroyed somehow?

For the 2 neurons (281525030204895507, 685705572931231473), when I try to disburse maturity, I get the error:
NNS error: Disbursement (0) is too small. After the worst case maturity modulation (-500) the amount should be at least: 100000000 e8s

@jasonzhu can you please help?

6

@Lorimer @jasonzhu I ran the disburse maturity for a third neuron - 721574941063530135.

quill neuron-manage 721574941063530135 --disburse-maturity --disburse-maturity-to d8ed3d2a3d4b302681ad992afc74ef757e98047b446bc9d87f23f1d85c585ce1 --pem-file ~/.config/dfx/identity/lockandkey/identity.pem > ./disburse-maturity-721574941063530135.json

quill send ./disburse-maturity-721574941063530135.json

and I got a success message

Successfully disbursed 275.87680771 maturity

However, this isn’t showing up in my account d8ed3d2a3d4b302681ad992afc74ef757e98047b446bc9d87f23f1d85c585ce1

Is there a waiting time or something else going on with the seed neurons?

7

Hi @idontexist, thanks for posting the steps you went through, those are really helpful to show what happened. From those it seems things worked as expected.

TLDR as you probably care most about the maturity in your neurons: there is a 7-day waiting period for both spawned neurons and disbursed maturity. After the 7-day waiting period, spawned neurons should exit the “spawning” state (and available as a dissolved neuron), while the disbursed maturity will arrive at the specified address

State of Spawning

While this is true for neurons controlled by NNS Dapp, it’s not true in general, and more specifically, not true for neurons controlled in other ways. The “spawning” is deprecated, but not removed from the NNS Governance (and thus it should still work correctly). In the NNS Dapp we present the easier way of accessing the maturity, but in other surfaces it’s more challenging to completely remove the spawning without breaking things, and the work to stop supporting spawning hasn’t happened yet.

Documentation

Please check this page (the section “Spawning maturity via the maturity modulation function”) for spawning and this page for disbursing maturity.

How to confirm things working as expected

For spawned neurons, the spawn_at_timestamp_seconds is the time when the neuron exit the “spawning” state. The timestamp shown for 9973304611228297795 seems to be the coming Sunday. At that time, you should see (1) cached_neuron_stake_e8s become ~33ICP (subject to maturity modulation of ±5%), and (2) spawn_at_timestamp_seconds becomes null. At that time you should be able to do quill neuron-manage --disburse ...

For disbursed maturity, you should be able to call

$ dfx canister --network ic call rrkah-fqaaa-aaaaa-aaaaq-cai get_full_neuron ‘(721574941063530135:nat64)’

and see a maturity_disbursements_in_progress field, and the finalize_disbursement_timestamp_seconds inside it should indicate when you should see the the minted ICP.

Hope this helps!

3 Likes
8

Oh wow thanks so much for the detailed explanation! I think I am all set then - totally missed the 7 day waiting period part. I’ll update after a week how things look then. Really appreciate your help, thanks once again!

A couple of questions if you don’t mind answering -

  • I have a few other seed neurons - is there a recommended way to access their staking rewards i.e. via reward spawning new neurons or direct disbursement to account? Does it matter?
  • I also noticed that my seed neurons aren’t participating in governance, which is very unfortunate. I thought these were automatically voting for ‘Dfinity’ back in Nov ‘21. I just wanted to ask and clarify - is it ok to get them voting now, even if some are awaiting the reward disbursement? And if so, what’s the best way to delegate votes to the Dfinity org or the recommended entity?
  • The seed neurons came with a whole dissolve delay schedule. I want to put them all to 4 years. Is it safe to just do this with the existing seed neurons without dissolving them? Any gotchas at all that I should be aware of?

Thanks again for your help!

9

I’d say that disbursing maturity is easier. However, we got report that Kraken has some problem handling such transactions and we are actively working with them to resolve this. In the mean time, I’d suggest disbursing to an address owned by your own principal.

You are probably affected by the periodic confirmation feature. This post has more information as well as the “Periodic Confirmation” section on this page.

I don’t see a problem of increasing the dissolve delay to 4 years. You are probably aware of the fact that if the state is not-dissolving, then the dissolve delay will remain 4 years instead of decreasing. Another thing is that the age bonus is lost once you start dissolving.

2 Likes
10

Hey @jasonzhu thanks for your responses above. I tried voting via quill per the post that you referenced in your earlier reply. The response I got seems different from what I should expect per that post. Is there a way to confirm this worked? I ran this for my first neuron (referenced in my original post) - 281525030204895507. Ideally, can I see this works now on a web UI somewhere to independently confirm?

$ quill neuron-manage 281525030204895507 --refresh-followers
[{"ingress":{"call_type":"update","request_id":"0c0bb0b62370e4a725335c822b1a71b68667b3b8e9a97f1cdcd18aa925af4a3a","content":"d9d9f7a167636f6e74656e74a76c726571756573745f747970656463616c6c656e6f6e63655023333b327609c4bad80623a4d30fd6966e696e67726573735f6578706972791b187b08dfe9c2d8186673656e64657241046b63616e69737465725f69644a000000000000000101016b6d6574686f645f6e616d656d6d616e6167655f6e6575726f6e6361726759071b4449444c8a016c03dbb70101cbe2b58b0803f1bb8b880d88016e026c01dbb701786e046b0f9b9cd0a40105bab5f1a40109918bacf1020a948c9eb4030ce387a8cf0314fc9fc6830515c6b3bb9106198db2d592092398a5d0c7092491b2fab80a25af88f5bb0b26e0f8fffd0b298bf3afac0d830189b8b3b30e8401a3f3c0ad0f85016c03bc949d820306dbe2be950907ef9999fe09086e796e686e786c01b9ef938008786c02afa3bda10175c2cee0d80c0b6d026c03ff8aae8b060da9ddf49b0710a290ead30a796e0e6c01cedfa0a8040f6d7b6e116c02b3b0dac30307ad86ca8305126e136c01ad86ca83050f6c006c01d7ab01166e176b039ef5cc0f149992ccd00118dae1c99903786c029cb1fa2507ba89e5c204786c01a78882820a1a6e1b6b0a96a7f7151cf381d4ab021d8cb2f18c071eb09b9ba40714d0fb87af071490f29afe071ff4f7d8e30b20e4ac938d0c14f7aacfd80d14c3a2f6c90e226c01f6b0989a08076c018eddc3a60d076c01d0e1e9f60c7e6c018dc3b2b303796c01d2dbb6980d216e756c01c88ecad50a786c02ea99cff20475b2b8d4960b016c01c38fbbd10b016c05f5bbe3900178d2bbf0d9017eb9ef93800878dbe2be950907ef9999fe09786c01c1ac9e9d06276d286c02afa3bda10121c2cee0d80c0b6c04efd6e4027198abec81012ab6f798b2012ba696a48708716e716e2c6b1193a7e09d022deef2d3b60330d881c9c40300f3a4c9c50432c8e7adc6053784b796cd073a88f0ccb4083bd69ce79d0add0082ffcfaa0cde009e9598a00de5009dfa94a40dea00e3c3c5990eef00b1a5aea10ef300f5d9d7a50ef500fad5ddf40efe00db9cebf70eff00d6f4c7ff0f82016c02dbb70101bac7a7fa0d2e6e2f6c02cbe4fdc70471fc91f4f8052a6c03cba4b6ed0407c9ef8ec50971bbf8fded0f316d686c02b3c4b1f20407e3f9f5d908336e346c07c0cff27108e8f09b7308d7e09b900235edd9c8c90721f8e287cc0c08deebb5a90e08a882acc60f086e366c01d7e09b9002316c07d6fca70238a79fc97e38dcd7f3a60439a6bd87b10438b3c4b1f2040797ab9ebc0538c7f0d6f80a216e0f6e7e6c02b6f798b20121b3c4b1f204076c0aefd6e4022ae7a390313cc1cbdfc70231ebbedebd04c600cbe4fdc7042ac0c9f2d804c800fc91f4f8052aaafababc09ca0096a8ca8a0ccc00e597bccc0dd5006e3d6c0aa0e2cd173ebeafa763c000c0a6dbaa01c00081b4c8f702c000f4ba8b81043ebdbbfabf06c200ebb6f3ff06c00091d7d7b007c00083fbafbc0bc200e2efc5a90fc4006e3f6c01c4d9d3ea0f086ec1006c01ffecb6d405086ec3006c01e0a9b302086ec5006c03ef9cc907c000f5eadfe7013ee7ce9ba50f3e6ec7006c01c3b9dfbf0c2a6ec9006c04a5d8d2a602c2009efeb9a4032ab1a38bf40dc60091c9aafe0d2a6dcb006c01dbb701076ecd006c0eb1f3893608c6f0968c0139d49eb8a401c000abe1b1d001ce00b7859b9d042aebaf95fd04c200a59ae0a605c200d3de91a907c200d9d3968b08c200caa0ddc708d00081deedcd08c200b7d0c9ff08c20085b1ad8e0cc200d08cf4c90fd2006ecf006c028fb2cf55c000cf898dd304086ed1006c01ceca9f900a086ed3006c01ece1e5a508d4006d716ed6006c03c09ea567d700d9edb29806d900f0abe7ef0cd7006ed8006c01c4a7c9a101c2006eda006c01ebb6bfc407db006ddc006c059cb1fa2507f5a1e557c000ba89e5c204088edf85ee05c000badcada008c2006c02e4d7bee905758effd6e90e0f6c03ce9ca6ce01df00f382ccb307e100b9ef938008786ee0006c02dbb701079df1afe7070d6ee2006b02fdf59aec0be300e3b586ff0ce4006c01f5bbe39001786c01a9ddf49b070d6c03d889bea60d08b5f6f9e90e07c6f6ebeb0ee6006ee7006c0bfed391bd0178abe1b1d001e800dcd0a0ab0378dfbcb4d80478edc6ecab087993e5e48109788ba7d38a0a0890b090af0e78f0a0acd20e08ca9ab7d20f789ea088f70f086ee9006c028fa0804178cf898dd304786c02cfbe93a404eb00c796cdbe0b076eec006c01edbb85f901ed006eee006c02f9889a5778b2cc99e705786c0184aead33f0006df1006c02007501f2006c01c2cee0d80c0b6c02f8b9b6c90439a4ccf7dd0af4006dde006c0a9eb493cf03788c98f1f603f600befa8dd40479be8fe6e30478ce89be97067886f998bc0978c5cae3d40a7893a190e00c78f5e1d0e70d78a3dc9a8d0ff8006ef7006c03c9c49fac020881a6cee60508bca4cff30c086ef9006c04c1868bdc053e86ffb7af07fa00a6c6f58f08fc0093a190e00c3e6efb006c0383b1b6c802fc0088e7f1c902fc0089dec1be0cfc006efd006c0182c0a0fa022a6c018594e2c50b316c01f0a2cabb0b80016e81016b02bf80e42be000c6a6e4b90ae0006c0196bdb4e904716c01bbb4b09703066c01b99d9da50b796c02a9ddf49b070dd8a38ca80d86016e87016c01e0a9b302786e89016b02cd8e8eb9040fcebee1d30802010000010401011371befc852de803","role":"nns:governance"},"request_status":{"canister_id":"rrkah-fqaaa-aaaaa-aaaaq-cai","request_id":"0c0bb0b62370e4a725335c822b1a71b68667b3b8e9a97f1cdcd18aa925af4a3a","content":"d9d9f7a167636f6e74656e74a46c726571756573745f747970656a726561645f73746174656e696e67726573735f6578706972791b187b08d2adefa8006673656e646572410465706174687381824e726571756573745f73746174757358200c0bb0b62370e4a725335c822b1a71b68667b3b8e9a97f1cdcd18aa925af4a3a"}}]
11

I believe the process should be similar to what you did to disburse maturity: quill neuron-manage 281525030204895507 --refresh-followers ... > ./refresh-followers.json and then quill send (I would do --dry-run before sending).

There are a few ways:

for neurons not controlled by an NNS Dapp principal, and you would be able to see information only readable by the neuron owner, without making the neuron public. Not sure what device you use so it’s hard to provide the exact instructions. Perhaps this video would be helpful if you use a Ledger device, but on a high level, you just need to add your NNS Dapp principal as a hotkey of the neuron (quill neuron-manage ${NEURON_ID} --add-hot-key ${YOUR_NNS_DAPP_PRINCIPAL} ... > add-hot-key.json and then quill send. You would need to have some level of trust on the NNS Dapp, of course.

1 Like
12

Thanks @jasonzhu ! I tried this but got an error:

$ quill send refresh-followers-281525030204895507.json --dry-run
Sending message with

  Call type:   update
  Sender:      2vxsx-fae
  Canister id: rrkah-fqaaa-aaaaa-aaaaq-cai
  Method name: manage_neuron
  Arguments:   (
  record {
    id = null;
    command = opt variant { RefreshVotingPower = record {} };
    neuron_id_or_subaccount = opt variant {
      NeuronId = record { id = 281_525_030_204_895_507 : nat64 }
    };
  },
)

but when I try to send, I get this error:

NNS error: The caller (2vxsx-fae) is not authorized to refresh the voting power of neuron 281525030204895507.

Tried adding my pem file location via --pem-file flag but get the same output. Could you help - seems like I am missing something here.

FWIW, I am not using a ledger device for this.

13

This is the problem - 2vxsx-fae is the anonymous principal, so essentially the quill neuron-manage produced a message signed by the anonymous principal (hence, not authorized). After looking at the quill implementation, I’m not seeing how the command would succeed with the anonymous principal if --pem-file is provided (as opposed to, fail if --pem-file is somehow invalid). Would it be possible that you created a refresh-followers-281525030204895507.json without --pem-file first, and then later used that previous file for sending?

Otherwise, it might be helpful to provide the exact version of quill so we can take a closer look. I also used quill to refresh my own neuron and it worked (I’m using 0.5.3).

1 Like
14

Ah, you’re absolutely right. Giving the –pem-file first worked - my bad here.

I’ll do this for all of my neurons. Just to confirm - I should be doing this once every 6 months right? So e.g. if I write a quick script to do this, I need to run this once every 6 months basically?

1 Like
15

Yes, that sounds right

1 Like
16

Thanks @jasonzhu really appreciate your help through some of these issues!