I think we should do something simple and flexible, which does not impose on the user how to exactly organize their security. This is why I like the first option here summarized by @timo :
creating two levels of authentication methods, higher level and lower level, and you need to be logged in with a higher level method in order to remove (i.e. delete) a lower level method. But
a) using the higher level must be optional,
b) the user must be able to choose which method to put on the higher level (for some it is the phrase, for others a Yubikey, etc.)
c) it must be possible to put more than one method to the higher level if the user wants that.
This is just like an admin vs user account in an OS. You can create multiple accounts of either type.
People that want to have a recovery phrase that can only be changed by entering it, could achieve that by setting the recovery phrase as the only device on the higher level.
Additionally, this simple model maps well to the current II backend, which does not differentiate between device types.