Issue with Node Provider Rewards

Dear Node Providers and ICP community,

As usual before the node provider reward distribution, we ran an analysis of the expected rewards and we identified an issue in the expected amounts of distributed rewards. The root cause of the issue is still unclear and we haven’t had time to fully diagnose the issue yet. Due to the high impact of the issue on the node rewards scheduled for distribution later today, we plan to take immediate corrective actions:

  • We’re submitting proposals to correct the registry data. This should fix the discrepancies in today’s reward payouts. We already know which node operator records are affected and expect these proposals to resolve the issue. Due to timing being critical, DFINITY Foundation will vote on them faster than usual, skipping the consideration period.
  • At the same time, our engineering teams are conducting a root-cause analysis to fully understand and fix the underlying problem.

We appreciate your patience while we work through this. We’ll share updates as soon as we have more information.

I you have any questions, we are happy to answer them.

5 Likes

Proposals to correct registry data:

4 Likes

Node provider rewards were issued successfully.

4 Likes

And here is the tentative fix for the root cause, pending reviews, merging, and deployment.

3 Likes

Proposal Reviews | Tim - CodeGov

Voted to adopt proposals 135283, 135284 and 135285. Temporary fix to ensure that node provider rewards are distributed correctly, already executed, with planned fix already pushed pending deployment.

About CodeGov

CodeGov has a team of developers who review and vote independently on the following proposal topics: IC-OS Version Election, Protocol Canister Management, Subnet Management, Node Admin, and Participant Management. The CodeGov NNS known neuron is configured to follow our reviewers on these technical topics. We also have a group of Followees who vote independently on the Governance and the SNS & Neurons’ Fund topics. We strive to be a credible and reliable Followee option that votes on every proposal and every proposal topic in the NNS. We also support decentralisation of SNS projects such as WaterNeuron, KongSwap, and Alice with a known neuron and credible Followees.

Learn more about CodeGov and its mission at codegov.org.

2 Likes

Proposal 135283, 135284 and 135285 – LaCosta | CodeGov

Vote: ADOPT

As explained by DFINITY there has been an issue with the distribution of the node rewards. The root cause of the issue is still being looked at but in the meanwhile this proposals were made for the Node Operators affected by the issue.

About CodeGov

CodeGov has a team of developers who review and vote independently on the following proposal topics: IC-OS Version Election, Protocol Canister Management, Subnet Management, Node Admin, and Participant Management. The CodeGov NNS known neuron is configured to follow our reviewers on these technical topics. We also have a group of Followees who vote independently on the Governance and the SNS & Neuron’s Fund topics. We strive to be a credible and reliable Followee option that votes on every proposal and every proposal topic in the NNS. We also support decentralization of SNS projects such as WaterNeuron, KongSwap, and Alice with a known neuron and credible Followees.

Learn more about CodeGov and its mission at codegov.org.

1 Like

Proposal #135283, #135284, #135285 Review — Louise | Aviate Labs

Vote: ADOPT
Review:

As explained by DFINITY, these proposals have been submitted and executed immediately to correct the issue regarding the distribution of node rewards. The root cause is still being investigated. These proposals were created for the affected Node Operators.

About Aviate Labs

Aviate Labs is a team dedicated to supporting node providers since 2020. Our mission is to make high-performance infrastructure management on the Internet Computer (ICP) as seamless as possible, while adhering to the principles of decentralization.

We are known for our contributions to the ecosystem, including the go-agent and developer work packages on GitHub, as well as the Node Monitor tool, which alerts Node Providers as soon as any of their nodes go down.

In the NNS, our team—Roald, Louise, and Quint—reviews and votes independently on ‘Node Admin’ and ‘Participant Management’ proposals. Currently we don’t vote at all on other proposals.

The Aviate Labs known neuron is configured to follow our team for these topics and other trusted entities for broader proposals. We strive to be a credible and reliable Followee, committed to voting on every proposal and supporting decentralization within the ICP ecosystem.

1 Like

Proposal #135283, #135284 & #135285 — Zack | CodeGov

Vote: Adopted

Reason:
As explained it is a FIX to correct registry data for Node Provider Rewards. As with every fix it was executed quickly but the update was provided and could be checked.

About CodeGov

CodeGov has a team of developers who review and vote independently on the following proposal topics: IC-OS Version Election, Protocol Canister Management, Subnet Management, Node Admin, and Participant Management. The CodeGov NNS known neuron is configured to follow our reviewers on these technical topics. We also have a group of Followees who vote independently on the Governance and the SNS & Neuron’s Fund topics. We strive to be a credible and reliable Followee option that votes on every proposal and every proposal topic in the NNS. We also support decentralization of SNS projects such as WaterNeuron, KongSwap, and Alice with a known neuron and credible Followees.

Learn more about CodeGov and its mission at codegov.org.

1 Like