Firewall rule changes on the IC

Hello, kind folks!

We are proposing to update the firewall rules for Orchestrator Dashboard, as currently, it lists old SF1, FR1 and CH1 prefixes.

It is possible to get the current firewall rules by running the following

❯ ic-admin --nns-urls https://ic0.app/ get-firewall-rules replica_nodes  | jq
...
  {
    "ipv4_prefixes": [],
    "ipv6_prefixes": [
      "2a00:fb01:400::/56",
      "2607:f6f0:3004::/48",
      "2001:4d78:40d::/48",
      "2607:fb58:9005::/48"
    ],
    "ports": [
      7070
    ],
    "action": 1,
    "comment": "Orchestrator Dashboard",
    "user": null,
    "direction": null
  },

This is the history of the prefixes change:

Cluster Old Prefix New Prefix Change Date
CH1 2607:f6f0:3004::/48 2602:fb2b:120::/48 13 Feb 2024
FR1 2001:4d78:40d::/48 2602:fb2b:110::/48 17 Jul 2024
SF1 2607:fb58:9005::/48 2602:fb2b:100::/48 5 Feb 2023

SF1 cluster has been migrated to DM1 on Jan 2025 but the new prefix remained the same

The proposed change is therefore the following:

REMOVE:
"2607:f6f0:3004::/48",
"2001:4d78:40d::/48",
"2607:fb58:9005::/48"

ADD:
"2602:fb2b:120::/48", 
"2602:fb2b:110::/48", 
"2602:fb2b:100::/48"

Do not hesitate to ask questions here!

5 Likes

Thanks @pietrodimarco! I take it this is the link relating to proposal 136670. (The link in the proposal did not work.)

In case anyone is not familiar with the Orchestrator, there’s some good documentation here.

4 Likes

@pietrodimarco I think I’ll need some further information before being able to vote “adopt” on this proposal.

I was able to run the command
ic-admin --nns-urls https://ic0.app get-firewall-rules replica_nodes | jq
after correcting a typo in the version above and installing jq. This produces a list of 60 ipv6 prefixes labeled “Firewall rules for all replica nodes” in addition to the 4 you’ve shown labeled “Orchestrator Dashboard” and some others. These 4 all appear in the “rules for all” list. I did a search around and found some general information on firewall rules but was not able to find any context specific to understanding the effects of making the change given in the proposal.

Do SF1, FR1, CH1 and DM1 refer to data centres or something else? None of these are codes for currently active data centres. Can you point us towards some documentation that explains how to interpret these rules? Is there any way for us to verify the changes to these prefixes?

Ideally I would want to vote on this before we get to the last 24 hours of the voting period. Apologies for the short time frame, as I had been focusing on reviewing other proposals before coming back to this one.

1 Like

Im surprised to see that this proposal was adopted by DFINITY much earlier than the deadline (more than 24 hours to go), and before any reviewer had posted a review.

I was planning to reject, as I see what looks like a major issue. I’m on my lunch break at the moment. I’ll vote to reject + post my review later after work.

2 Likes

Proposal 136670 | Tim - CodeGov

Vote: Reject

This proposal is for changes to the firewall rules, but requires further context to be given in order for an informed vote to be cast. I am also unaware of any way to verify the contents of the proposal payload.

About CodeGov

CodeGov has a team of developers who review and vote independently on the following proposal topics: IC-OS Version Election, Protocol Canister Management, Subnet Management, API Boundary Node Management, Node Admin, and Participant Management. The CodeGov NNS known neuron is configured to follow our reviewers on these technical topics. We also have a group of Followees who vote independently on the Governance and the SNS & Neurons’ Fund topics. We strive to be a credible and reliable Followee option that votes on every proposal and every proposal topic in the NNS. We also support decentralisation of SNS projects such as WaterNeuron, KongSwap, and Alice with a known neuron and credible Followees.

Learn more about CodeGov and its mission at codegov.org.

1 Like

Proposal 136670 Review | Malith H - CO.DELTA △

VOTE: NO :cross_mark:

TLDR:
The proposal is to update the IPv6 prefix of Orchestrator Dashboard Rule with

2602:fb2b:120::/48,
2602:fb2b:110::/48,
2602:fb2b:100::/48

:warning: Issue:

Orchestrator Dashboard is rule 3 with port 7070 updated - 2022-08-18
The current proposal does not update the IPv6 prefix but adds it to position 2, making port 7070 redundant in rules 2 and 3

After discussing with team mates @Lorimer @aligatorr89 we have come to an independent conclustion to reject. Issue was first picked up by @Lorimer

Firewall rule update history

Date Position Comment Ports IPv6 Prefixes
2022-08-18 3 :warning: Orchestrator Dashboard 7070 2a00:fb01:400::/56, 2607:f6f0:3004::/48, 2001:4d78:40d::/48, 2607:fb58:9005::/48
2023-01-16 0 Firewall rules for all replica nodes 22, 2497, 4100, 8080, 9090, 9091, 9100, 19531 59 prefixes including: 2001:438:fffd:11c::/64, 2607:fb58:9005::/48, 2a00:fb01:400::/56, fd00:2:1:1::/64
2024-02-29 5 SH1 boundary nodes 8080 2001:4c08:2003:b09::/64
2024-03-15 0 Firewall rules for all replica nodes 22, 2497, 4100, 8080, 9090, 9091, 9100, 19100, 19531 Same 59 prefixes as 2023-01-16
2024-05-04 5 LN1 boundary nodes 8080 2a0b:21c0:4003:1::/64

You may wish to follow the CO.DELTA known neuron if you found this analysis helpful.

CO.DELTA

We’re a verifiably decentralised collective who review IC deltas (changes applied by NNS proposals). We follow a common code:

  • Look: We observe the details and context of NNS proposals
  • Test: We test and verify the claims made by those proposals
  • Automate: We automate as much as possible by building increasingly sophisticated tools that streamline and strengthen the reviewal process.

Every vote cast by CO.DELTA is the result of consensus among diligent, skilled and experienced team members acting independently. The CO.DELTA neuron follows the vote of D-QUORUM on NNS topics that the CO.DELTA team does not handle directly. You can therefore follow CO.DELTA on all topics and rely on the highest quality of vote.

2 Likes

Proposal #136670 — Zack | CodeGov

Vote: Adopted

Reason:
The proposal updates the firewall rules for Orchestrator Dashboard, by removing :
"2607:f6f0:3004::/48", "2001:4d78:40d::/48", "2607:fb58:9005::/48"
and adding :
"2602:fb2b:120::/48", "2602:fb2b:110::/48", "2602:fb2b:100::/48"
to the existing "2a00:fb01:400::/56".

Update : As can bee seen this is exactly what was executed. So the proposal was correct.

About CodeGov

CodeGov has a team of developers who review and vote independently on the following proposal topics: IC-OS Version Election, Protocol Canister Management, Subnet Management, Node Admin, and Participant Management. The CodeGov NNS known neuron is configured to follow our reviewers on these technical topics. We also have a group of Followees who vote independently on the Governance and the SNS & Neuron’s Fund topics. We strive to be a credible and reliable Followee option that votes on every proposal and every proposal topic in the NNS. We also support decentralization of SNS projects such as WaterNeuron, KongSwap, and Alice with a known neuron and credible Followees.

Learn more about CodeGov and its mission at codegov.org.

Proposal 136670 Review | Lorimer - CO.DELTA △

VOTE: NO

TLDR: The proposed prefixes are sure enough owned by DFINITY, but this proposal has accidentally overwritten a separate ‘Allow’ firewall rule for the “DFINITY monitoring stack”, while leaving a redundant Orchestrator firewall rule at a lower priority level (higher position).

Therefore, while this proposal had the desired effect of updating the effective firewall for the Orchestrator on replica nodes (due to inserting a rule at a lower position with higher priority), it left the firewall rule that it intended to update untouched, and instead overwrote a separate rule for separate pieces of infrastructure. Given that it was an ‘Allow’ rule that was overwritten, I would suspect this means that various parts of the DFINITY monitoring stack have now lost visibility of the replica nodes they were monitoring (unless I’m mistaken about the ramifications).

@pietrodimarco am I correct in this analysis, or have I got the wrong end of the stick? Can I ask why did DFINITY adopted this proposal before any reviews were posted? There’s still quite a long time left in the voting period.

Details

Prior to this proposal, the most recent proposal that updated the ReplicaNodes-scoped firewall rule at position 2 was this one → Proposal 74396 in 2022 (given that Proposal 128170 never executed).

As highlighted by @MalithHatananchchige in his review above, the ReplicaNode-scoped firewall rule for the Orchestrator is at position 3 (not 2, as expected by this proposal). It was added by Proposal 76092 in 2022.

By updating the rule at position 2, what this proposal instead appears to have done is overwrite the effects of Proposal 74396, while the intention was to overwrite the effects of Proposal 76092. You can see this in the code →

and given that this proposal has nonetheless executed, you can see this by running ic-admin --nns-urls https://ic0.app/ get-firewall-rules replica_nodes | jq (the “DFINITY monitoring stack” rule has disappeared)

Confirming Prefix Ownership

In any case, the prefixes ‘Allowed’ by this proposal belong to DFINITY. DFINITY claim these are their ‘InfraDCs’ for running infrastructure monitoring/maintenance tooling. I don’t see this as a claim that needs verifying, beyond the fact that DFINITY control these servers. After all, they can do what they want with the access they have. In an ideal world though, one day Node Providers should be capable of monitoring their own nodes (and managing CUPs when needed and what not).


You may wish to follow the CO.DELTA known neuron if you found this analysis helpful.

CO.DELTA △

We’re a verifiably decentralised collective who review IC deltas (changes applied by NNS proposals). We follow a common code:

  • Look: We observe the details and context of NNS proposals
  • Test: We test and verify the claims made by those proposals
  • Automate: We automate as much as possible by building increasingly sophisticated tools that streamline and strengthen the reviewal process.

Every vote cast by CO.DELTA is the result of consensus among diligent, skilled and experienced team members acting independently. The CO.DELTA neuron follows the vote of D-QUORUM on NNS topics that the CO.DELTA team does not handle directly. You can therefore follow CO.DELTA on all topics and rely on the highest quality of vote.

3 Likes

Proposal 136670 Review | aligatorr - CO.DELTA △

VOTE: YES

TLDR: Proposal changes firewall rules as stated:

  • “2607:f6f0:3004::/48” → “2602:fb2b:120::/48”
  • “2001:4d78:40d::/48” → “2602:fb2b:110::/48”
  • “2607:fb58:9005::/48” → “2602:fb2b:100::/48”
Firewal Rules Before
[
    {
        "ipv4_prefixes": [],
        "ipv6_prefixes": [
            "2001:4c08:2003:b09::/64",
            "2001:438:fffd:11c::/64",
            "2001:470:1:c76::/64",
            "2001:4d78:400:10a::/64",
            "2001:4d78:40d::/48",
            "2001:920:401a:1706::/64",
            "2001:920:401a:1708::/64",
            "2001:920:401a:1710::/64",
            "2401:3f00:1000:22::/64",
            "2401:3f00:1000:23::/64",
            "2401:3f00:1000:24::/64",
            "2600:2c01:21::/64",
            "2600:3000:1300:1300::/64",
            "2600:3000:6100:200::/64",
            "2600:3004:1200:1200::/56",
            "2600:3006:1400:1500::/64",
            "2600:c02:b002:15::/64",
            "2600:c0d:3002:4::/64",
            "2602:fb2b::/36",
            "2602:ffe4:801:16::/64",
            "2602:ffe4:801:17::/64",
            "2602:ffe4:801:18::/64",
            "2604:1380:4091:3000::/48",
            "2604:1380:40e1:4700::/48",
            "2604:1380:40f1:1700::/64",
            "2604:1380:45d1:bf00::/64",
            "2604:1380:45e1:a600::/48",
            "2604:1380:45f1:9400::/64",
            "2604:1380:4601:6200::/48",
            "2604:1380:4641:6100::/48",
            "2604:3fc0:2001::/48",
            "2604:3fc0:3002::/48",
            "2604:6800:258:1::/64",
            "2604:7e00:30:3::/64",
            "2604:7e00:50::/64",
            "2604:b900:4001:76::/64",
            "2607:f1d0:10:1::/64",
            "2607:f6f0:3004::/48",
            "2607:f758:1220::/64",
            "2607:f758:c300::/64",
            "2607:fb58:9005::/48",
            "2607:ff70:3:2::/64",
            "2610:190:6000:1::/64",
            "2610:190:df01:5::/64",
            "2a00:fa0:3::/48",
            "2a00:fb01:400:200::/64",
            "2a00:fb01:400::/56",
            "2a00:fc0:5000:300::/64",
            "2a01:138:900a::/48",
            "2a01:2a8:a13c:1::/64",
            "2a01:2a8:a13d:1::/64",
            "2a01:2a8:a13e:1::/64",
            "2a02:418:3002:0::/64",
            "2a02:41b:300e::/48",
            "2a02:800:2:2003::/64",
            "2a04:9dc0:0:108::/64",
            "2a05:d01c:e2c:a700::/56",
            "2a0b:21c0:b002:2::/64",
            "2a0f:cd00:0002::/56",
            "fd00:2:1:1::/64"
        ],
        "ports": [
            22,
            2497,
            4100,
            8080,
            9090,
            9091,
            9100,
            19100,
            19531
        ],
        "action": 1,
        "comment": "Firewall rules for all replica nodes",
        "user": null,
        "direction": null
    },
    {
        "ipv4_prefixes": [],
        "ipv6_prefixes": [
            "2a0b:21c0:4003:2::/64"
        ],
        "ports": [
            8080
        ],
        "action": 1,
        "comment": "LN1 boundary nodes",
        "user": null,
        "direction": null
    },
    {
        "ipv4_prefixes": [],
        "ipv6_prefixes": [
            "2a00:fb01:400::/56",
            "2607:f6f0:3004::/48",
            "2001:4d78:40d::/48",
            "2607:fb58:9005::/48"
        ],
        "ports": [
            7070
        ],
        "action": 1,
        "comment": "Orchestrator Dashboard",
        "user": null,
        "direction": null
    },
    {
        "ipv4_prefixes": [],
        "ipv6_prefixes": [
            "2600:c00:2:100::/64"
        ],
        "ports": [
            8080
        ],
        "action": 1,
        "comment": "SE1 boundary nodes",
        "user": null,
        "direction": null
    },
    {
        "ipv4_prefixes": [],
        "ipv6_prefixes": [
            "2001:4c08:2003:b09::/64"
        ],
        "ports": [
            8080
        ],
        "action": 1,
        "comment": "SH1 boundary nodes",
        "user": null,
        "direction": null
    },
    {
        "ipv4_prefixes": [],
        "ipv6_prefixes": [
            "2a0b:21c0:4003:1::/64"
        ],
        "ports": [
            8080
        ],
        "action": 1,
        "comment": "LN1 boundary nodes",
        "user": null,
        "direction": null
    }
]
Firewal Rules After
[
    {
      "ipv4_prefixes": [],
      "ipv6_prefixes": [
        "2001:4c08:2003:b09::/64",
        "2001:438:fffd:11c::/64",
        "2001:470:1:c76::/64",
        "2001:4d78:400:10a::/64",
        "2001:4d78:40d::/48",
        "2001:920:401a:1706::/64",
        "2001:920:401a:1708::/64",
        "2001:920:401a:1710::/64",
        "2401:3f00:1000:22::/64",
        "2401:3f00:1000:23::/64",
        "2401:3f00:1000:24::/64",
        "2600:2c01:21::/64",
        "2600:3000:1300:1300::/64",
        "2600:3000:6100:200::/64",
        "2600:3004:1200:1200::/56",
        "2600:3006:1400:1500::/64",
        "2600:c02:b002:15::/64",
        "2600:c0d:3002:4::/64",
        "2602:fb2b::/36",
        "2602:ffe4:801:16::/64",
        "2602:ffe4:801:17::/64",
        "2602:ffe4:801:18::/64",
        "2604:1380:4091:3000::/48",
        "2604:1380:40e1:4700::/48",
        "2604:1380:40f1:1700::/64",
        "2604:1380:45d1:bf00::/64",
        "2604:1380:45e1:a600::/48",
        "2604:1380:45f1:9400::/64",
        "2604:1380:4601:6200::/48",
        "2604:1380:4641:6100::/48",
        "2604:3fc0:2001::/48",
        "2604:3fc0:3002::/48",
        "2604:6800:258:1::/64",
        "2604:7e00:30:3::/64",
        "2604:7e00:50::/64",
        "2604:b900:4001:76::/64",
        "2607:f1d0:10:1::/64",
        "2607:f6f0:3004::/48",
        "2607:f758:1220::/64",
        "2607:f758:c300::/64",
        "2607:fb58:9005::/48",
        "2607:ff70:3:2::/64",
        "2610:190:6000:1::/64",
        "2610:190:df01:5::/64",
        "2a00:fa0:3::/48",
        "2a00:fb01:400:200::/64",
        "2a00:fb01:400::/56",
        "2a00:fc0:5000:300::/64",
        "2a01:138:900a::/48",
        "2a01:2a8:a13c:1::/64",
        "2a01:2a8:a13d:1::/64",
        "2a01:2a8:a13e:1::/64",
        "2a02:418:3002:0::/64",
        "2a02:41b:300e::/48",
        "2a02:800:2:2003::/64",
        "2a04:9dc0:0:108::/64",
        "2a05:d01c:e2c:a700::/56",
        "2a0b:21c0:b002:2::/64",
        "2a0f:cd00:0002::/56",
        "fd00:2:1:1::/64"
      ],
      "ports": [
        22,
        2497,
        4100,
        8080,
        9090,
        9091,
        9100,
        19100,
        19531
      ],
      "action": 1,
      "comment": "Firewall rules for all replica nodes",
      "user": null,
      "direction": null
    },
    {
      "ipv4_prefixes": [],
      "ipv6_prefixes": [
        "2a0b:21c0:4003:2::/64"
      ],
      "ports": [
        8080
      ],
      "action": 1,
      "comment": "LN1 boundary nodes",
      "user": null,
      "direction": null
    },
    {
      "ipv4_prefixes": [],
      "ipv6_prefixes": [
        "2a00:fb01:400::/56",
        "2602:fb2b:120::/48",
        "2602:fb2b:110::/48",
        "2602:fb2b:100::/48"
      ],
      "ports": [
        7070
      ],
      "action": 1,
      "comment": "Orchestrator Dashboard",
      "user": null,
      "direction": null
    },
    {
      "ipv4_prefixes": [],
      "ipv6_prefixes": [
        "2600:c00:2:100::/64"
      ],
      "ports": [
        8080
      ],
      "action": 1,
      "comment": "SE1 boundary nodes",
      "user": null,
      "direction": null
    },
    {
      "ipv4_prefixes": [],
      "ipv6_prefixes": [
        "2001:4c08:2003:b09::/64"
      ],
      "ports": [
        8080
      ],
      "action": 1,
      "comment": "SH1 boundary nodes",
      "user": null,
      "direction": null
    },
    {
      "ipv4_prefixes": [],
      "ipv6_prefixes": [
        "2a0b:21c0:4003:1::/64"
      ],
      "ports": [
        8080
      ],
      "action": 1,
      "comment": "LN1 boundary nodes",
      "user": null,
      "direction": null
    }
  ]

You may wish to follow the CO.DELTA known neuron if you found this analysis helpful.

CO.DELTA

We’re a verifiably decentralised collective who review IC deltas (changes applied by NNS proposals). We follow a common code:

  • Look: We observe the details and context of NNS proposals.
  • Test: We test and verify the claims made by those proposals.
  • Automate: We automate as much as possible by building increasingly sophisticated tools that streamline and strengthen the reviewal process.

Every vote cast by CO.DELTA is the result of consensus among diligent, skilled and experienced team members acting independently. The CO.DELTA neuron follows the vote of D-QUORUM on NNS topics that the CO.DELTA team does not handle directly. You can therefore follow CO.DELTA on all topics and rely on the highest quality of vote.

2 Likes

@Lorimer I just ran the following command which confirms that the firewall rules have been correctly updated. Which command did you run when you saw different rules updated?

❯ ic-admin --nns-urls https://ic0.app/ get-firewall-rules replica_nodes  | jq
[...]
  {
    "ipv4_prefixes": [],
    "ipv6_prefixes": [
      "2a00:fb01:400::/56",
      "2602:fb2b:120::/48",
      "2602:fb2b:110::/48",
      "2602:fb2b:100::/48"
    ],
    "ports": [
      7070
    ],
    "action": 1,
    "comment": "Orchestrator Dashboard",
    "user": null,
    "direction": null
  },
[...]

There have been multiple other firewall proposals executed, that changed the firewall rule chain and moved the Orchestrator Dashboard up the chain (from position 3 to 2). E.g. this one.

We are following up internally to see if the orchestrator dashboard can be made public. IMHO there is nothing sensitive in it, and it can help for proposal / subnet recovery process to have the dashboard public.

5 Likes

Thanks @Sat, I failed to account for that removal proposal which executed last year. This proposal makes sense now. Apologies for the false alarm :slightly_smiling_face:

Given that ic-admin doesn’t show you firewall position information in the get-firewall-rules output, I don’t believe the get-firewall-rules output is sufficient for verifying proposals of this sort. It appears to show you the effective firewalls (having applied the rules), rather than the rules themselves (which reside a different levels of priority - i.e. it’s possible for one rule to hide another). Either this, or ic-admin isn’t providing all of the necessary information about each rule.

How are you supposed to verify updates to firewall rules which are applied at specific positions (and which would overwrite whatever else lives at that position)? If it hadn’t been for that firewall removal proposal, this proposal would have had the effect that me and @MalithHatananchchige surmised (unless I’m still mistaken).

I’m in the process of building a utility that collates firewall update history to establish the current state (including position information). This should avoid the problem of failing to spot all of the relevant history manually.

Can I ask how DFINITY go about checking which rules live at which position?

3 Likes

IIUC the firewall rules are just a list of dicts, and the position is the offset in the list, starting from zero. Or did I misunderstand the question?

position isn’t just where the rule is stored in the vector – it represents the rule’s actual priority.

These are translated into nftables (or iptables) using the same order, and Linux firewalls use the first match (such that later rules are not evaluated).

In any case, even if there weren’t priority attached to position (though there is :slightly_smiling_face:) it would still be the case that updates at a specific position would overwrite whatever is already there (so it’s important to verify what currently resides at that location when reviewing a proposal of this sort).

@ZackDS, you seemed to be fairly confident in your post. Can I ask how you verified the correctness of the firewall position?


@pietrodimarco, when you submitted this proposal, how did you ascertain that position 2 should be used? This will help with identifying other approaches that reviewers can use. Thanks