The NNS dapp is merely a frontend and the identity anchor you use to log in into this frontend is meaningless for the purpose of controlling ICP that are on the Ledger. The login is only there because the NNS dapp happens to provide other features that actually use the identity.
Nothing happens to the ICP that are on the Ledger if you lose control of that identity. You can just log in with a different identity, connect the hardware wallet and the coins are there. Conversely, if someone else gains control of the identity then without the Ledger they cannot control your coins that are on the Ledger.