CodeGov reviewed and voted to adopt the proposals from this batch, except for one that was rejected. Happy Holidays
3 Likes
It turned out that CodeGov had two out of three reviewers who felt they had time to perform the reviews. For a short period I wasnât sure if that could happen. I would have also been very supportive if they had decided to reject due to a lack of time. In this case, I think our two reviewers who didnât have other commitments were willing to review because they were aware that DFINITY had indicated that removal of the cordoned nodes was a priority for DFiNITY. However, there were questions about whether or not it was truly a critical emergency. My impression is that it can be ok to submit these proposals close to a holiday, but it would be best to verify that known reviewers are available first. Thatâs probably the part I would recommend changing in the future.
I have recently observed some folks in the community getting frustrated when they see so many proposals in their Actionable Proposals list in the NNS dApp and their Followee selection (e.g. CodeGov) hasnât voted yet when there is 0-2 days left in the voting period. In order to build a reputation as a credible and reliable Followee option, it is my opinion that just doing a good job is not enough. We also have to do it in a timely manner. Otherwise people can start to feel like they need to pencil whip their votes without reviews. I prefer to see the CodeGov team finish our reviews within 48 hours or as close as practical. Hence, in my opinion, the last day is simply too lateâŚespecially when there are 20 proposals in the que.
I asked the CodeGov team how long it takes them for each Subnet Management reviews. One reviewer said each proposal takes 15 minutes if there is 1 node change, 25 minutes if there are 2 node changes, and 30 minutes if the effect on the degree of decentralization is difficult. Another reviewer said the previous batch of 20 proposals took more than 3 hours and when there are large batches like this then he is less likely to use ic-admin to double check topology parameters. Our other reviewer indicated that he probably spent around 2 hours on the previous batch of 20 proposals. My conclusion from this feedback from @ZackDS @timk11 and @LaCosta and the feedback from @Lorimer continues to be that this proposal topic is under funded by a significant amount. There are simply too many proposals for the amount that the reviewers are getting paid. Iâm very concerned that this will get worse over the next several months due to the interim remuneration shuffling that is happening for Gen1 node providers.
Anyway, wishing you a Merry Christmas. I hope youâve had a great day @sat.
3 Likes
Agree that we need to look into the time spent on these proposals. I reviewed all the proposals myself, I estimate it took my about an hour excl. looking into details when there is discussion on proposals or questions. But my focus has been primarily whether the nodes are indeed excess nodes as I probably have good visibility on it; I agree for community review and going into details on DC location/triangulation/discussing open questions I can imagine it taking more time e.g. 2 hours. From my perspective, all proposals are okay to adopt, with 134579 debatable (reducing the decentralization on DC level) but with @sat clarification and intended follow-up proposal I do not see any major objections.
134563 replacing a dead node from NP Karel Frank, and replacing a nodes from NP BLP22 to optimize decentralization
134564 cordoning an excess node from NP Sygnum in Switzerland
134565 cordoning an excess node from NP Starbase in Japan
134566 cordoning excess nodes from NP 162Capital in Singapore and from NP Starbase in Japan
134567 cordoning an excess node from NP Starbase in Japan
134570 cordoning an excess node from NP BlockchainDevelopment Labs in Canada
134571 cordoning excess nodes from NP 162Capital in Singapore and from NP BlockchainDevelopment Labs in Canada
134572 cordoning an excess node from Rivonia Holdings in US
134573 cordoning excess nodes from NP Allusion in Belgium and from NP Starbase in Japa
134574 cordoning an excess node from NP Starbase in Japan
134575 cordoning excess nodes from NP Allusion in Belgium and from NP Starbase in Japan
134576 cordoning excess nodes from NP BlockchainDevelopment Labs in Canada and from NP Starbase in Japan
134578 cordoning an excess node from NP Dfinity in France
134579 cordoning excess nodes from NP 162Capital in Singapore and from NP Dfinity in US
134580 cordoning an excess node from Rivonia Holdings in US
134581 cordoning excess nodes from NP Allusion in Belgium and from NP Dfinity in US
134582 cordoning excess nodes from NP BlockchainDevelopment Labs in Canada and from NP Allusion in Belgium
134583 cordoning excess nodes from NP MI Servers in US, NP BlockchainDevelopment Labs in Canada and from NP Allusion in Belgium
Regarding the timing of the proposals, although it is holiday period I support to submit these proposals as the Gen1 Node Providers need enough time to offboard there nodes and hand them over to new node providers. So the earlier these nodes are cordoned, the better. Also, these Node Providers are submitting proposals themselves during the holiday period in preparation for this that require review. But surely agree with @zackds and @lorimer it is good to discuss honest review timing and the procedures for review during holiday period.
3 Likes
It sounds like it took you 3 minutes per proposal. Aside from what you focused on, did you review everything you think needs reviewing for a subnet membership proposal?
Note that a 2 hour total weekly estimate was put forward by DFINITY several months ago, back when the number of proposals were significantly fewer in number.
2 Likes
Hi @Lorimer yes I reviewed everything but keep in mind that I have the overview of all Gen1 Node Providers with excess nodes as the team has been aligning with them regarding the excess nodes. That is why in the above post I said I agreed with @ZackDS that reviewing the proposals takes more time for a known neuron/community member, and can easily takes 1.5 hour or more, in particular when there is discussion on proposals. And I did not include discussion and responses on the forum (like this one 
)
3 Likes
6 Likes
Do you consider the original 2 hour estimate (based on periods of significantly fewer proposal submissions) to have been an overestimate? Otherwise, Iâm unclear why you think more proposals overall can be handled in less time overall.
This is an interesting way of putting it. Could you clarify what you mean by âeverythingâ. Listing out the things that were checked would be useful. I think this is the only way to make sense of the time commitment you quoted.
New subnet management proposals have been submitted. Links to reviews should appear under this post shortly.
Just to help further the discussion on time requirements, I kept track of the time I spent on the current group of 5 non-critical subnet management proposals (134619 & 134633-6). This added up to 65 minutes, so 13 minutes per proposal on average, comprising checking details and writing reviews (53 mins), voting (4 mins) and posting the reviews in the forum along with copying the review links to the shared spreadsheet (8 mins).
Note that I took some shortcuts in that I only checked the topology parameters with the DRE tool for one of these proposals, and for proposal 134636 I relied on having already checked forum links for offboarding some of the data centres while reviewing earlier proposals. I feel my time management for this exercise was reasonably efficient (helped by the fact I was timing myself) with no significant time wastage.
4 Likes
Thanks @timk11, I liked your reviews. Timing them is a great idea. Maybe it would be a good idea to include these on each review in the future? @marc0olo, you indicated that this may be a useful requirement to add previously.
Have there been any more thoughts on this?
Other things to check include:
- Does the node being added already belong to the subnet?
- Is the node being added also being added by another open proposal?
- Is the node being removed the same node thatâs being added?
- Will the subnet still have at least one DFINITY node after the proposal has executed?
- Does the proposal follow prior agreed procedure (this differs based on the reasoning in the proposal summary)
- Are there extra bits of information not provided in the proposal that need to be sought out? As @timk11 mentioned, I did this in my reviews which later reviews could use to find the same information
- Checking decentralisation metrics for each proposal and cross-referencing with:
- the claims in the proposal summary, and
- the IC Target Topology
- Optionally checking that IP address geolocation agrees with the nodeâs claimed location (which is the only means of verifying this). Note that the decentralization metrics rest on the claimed location being accurate
- Anything else that is thrown up by the specifics of the proposal at hand (there are more nuanced things I could list out, such as is a node being added to a subnet in the list of cordoned nodes)
This canât be done in 3 minutes. It takes much closer to the time that @timk11 quoted, or slightly longer.
Thanks for the feedback @Lorimer . Youâve made some good points, but I probably wouldnât be in favour of recording time spent routinely or as a mandatory requirement. Depending on oneâs situation itâs sometimes hard to find a chunk of time to do these reviews free of interruptions or distractions, so estimating actual time spent can be tricky. There might also be background reading or other tasks where itâs not clear if this should be included in the tracking. On this occasion I stayed up late specifically to avoid these issues and to make sure I could do an accurate timing. It might still be useful for reviewers to report time spent on the odd occasion, particularly when these sorts of discussions are taking place. As you mentioned, there are several other aspects that could also be checked for a more thorough review, so my figures could well be underestimating the time that could reasonably be spent on these proposals.
3 Likes
I agree. I would not want to see a mandatory requirement to record time spent on these reviews. Nobody wants to punch a time clock or have their time micromanaged. We just need enough empirical data to understand the average time commitment. Hopefully that is enough to re-evaluate the grant size for this topic moving forward.
1 Like
@NoviSystems / Cody - Just wanting to check your intentions with regards to your node machines in data centre FM1. Proposal 134664 proposes to remove 106 nodes in 4 data centres from the network. Yours are the only ones that appear on the dashboard as âAwaitingâ rather than âOfflineâ, so I wasnât sure if yours were in a different situation from the other nodes in this DC.
@ZackDS @LaCosta @louisevelayo @quint @roald-av8 @wpb @sat @katiep
4 Likes
Thank you for bringing this to my attention. I think these nodes may have been included in the proposal in error; they are owned and operated by NoviSystems, LLC free and clear of any other obligation or contract. At this time they have not been offboarded, and there is no intention of removing them from the network. Thank you for catching this @timk11 before the proposal went through and theyâre removed.
For reference, the node IDs are:
- clgez-cnier-zieyq-77d3d-njbsf-p34cx-pbzm4-stdmo-wy62k-d5ng3-4ae
- fdunq-3nknr-3nex5-eravm-bauyc-ry3fs-mncyc-nzp4b-4prvk-4grhz-yae
- gojj3-3h3jg-rm33m-ces4z-bzops-c42hx-kwpjy-vm2ev-bfpzk-ednuh-gqe
(Lines 39, 41, 42)
5 Likes
Proposal 134664 Review | Louise - Aviate Labs
Vote: REJECT
Review:
- As highlighted by @timk11 (good catch
) and confirmed by @NoviSystems , 3 nodes will be mistakenly be removed from the registry. - While the rest of the details of the proposal are correct and justified by this post, I vote to reject to avoid the unnecessary removal of the 3 nodes mentioned above.
4 Likes
Hey @NoviSystems would you please clarify further if you are aware that the data center where these 3 nodes are located is planned to be decommissioned as a data center for the IC since it was originally established by Dfinity? When is your current contract with the physical data center set to expire? If the current data center is deprecated as planned and you plan to remain in the current location, then when do you plan to submit the new proposal to establish a new data center for the IC? Iâm trying to better understand how your 3 nodes could have been missed in relation to the long established plans to remove all nodes and decommission the data center.
1 Like
@wpb, no, I was unaware of these plans when I originally onboarded nodes there over a year ago. Per step 9 of the node provider onboarding document - if a record already exists for the datacenter nodes will be operated in, the step to create a new record can be skipped. As such I reused the existing record.
If there are still active nodes there, why does the data center need to be removed from the network at all? I have no plans of creating a proposal to create a new data center because until this morning I did not realize this would be an issue. What is the reason the entire datacenter record itself must be removed rather than just the affected nodes?
The contract I have with the facility is indefinite and not specific to these IC nodes. Are you proposing I create a new DC record in the exact same spot and go through the process of transferring my nodes there?
3 Likes
Thank you for the further clarification @novisystems . I think we will need further clarification from @katiep @SvenF @sat. Itâs entirely possible that this proposal needs to be rejected based on your situation, but before CodeGov and Aviate Labs both cast additional votes to reject, Iâm hoping to get further input from Dfinity. I suggest holding off on any further actions until we learn more about why these 3 nodes are included in this proposal.
1 Like
Proposal 134664 | Tim - CodeGov
Vote: Reject
This proposal is intended to remove 106 âoffboardedâ nodes from 4 data centres. I verified using the decentralization tool that the 106 nodes slated for removal do indeed correspond to the full set of nodes in the 4 data centres mentioned. However, 3 of the nodes in data centre FM1 appear as still online / âUnassignedâ and belonging to @NoviSystems . From the discussion in the forum above and in Matrix, it appears that these nodes are unrelated to the 4-year arrangement that affects the other nodes in this DC, and that Dfinity did not realise that these had been added at a later date. Hence I have voted to reject the proposal as these nodes were included in error and the node provider has expressed a desire for these nodes to remain in place.
About CodeGovâŚ
CodeGov has a team of developers who review and vote independently on the following proposal topics: IC-OS Version Election, Protocol Canister Management, Subnet Management, Node Admin, and Participant Management. The CodeGov NNS known neuron is configured to follow our reviewers on these topics and Synapse on most other topics. We strive to be a credible and reliable Followee option that votes on every proposal and every proposal topic in the NNS. We also support decentralization of SNS projects such as WaterNeuron and KongSwap with a known neuron and credible Followees.
Learn more about CodeGov and itâs mission at codegov.org.
3 Likes