For whatever reason if there is a mismatch in root hash of the state tree in one of the nodes in the subnet after the execution of the transactions in the block, what is the remedial action taken? Perhaps this can happen because some canister panicked and its state was not completely updated in the StateTree. Thanks in advance.
The node will crash and resume from the last checkpoint known to be good (i.e., agreed upon). If this checkpoint is fresh enough so that the blocks since then are still available, consensus will redeliver the blocks so that the node can catch up again with respect to the most recent height. If not, the node will initiate a state sync to obtain the most recent checkpoint that is known to be agreed upon and rejoin this way.
The reasons why this can happen is either a bug or some problems of one individual node, e.g., a broken disk.
Note that a panic of a canister can not lead to divergence. If a canister panics it happens deterministically and the protocol makes sure that all nodes consistently update (or roll back in this case) the canister’s state.