Let’s make some analogies before you ditch II entirely 
. How much money would you store in or access through MetaMask, MEW, etc. (without connecting a hardware wallet to them)? About the same you can store in II. Both depend on browser security. If the browser/OS is compromised then you can lose funds.
The point I made is that II is a software wallet. Software wallets have the advantage that they can store key material and you don’t have to confirm every single interaction on an external device and display. But if the browser/OS is compromised then the key material is at risk. Hardware wallets protect against that at the cost of having to approve every single interaction on an external device and display. Fundamentally there are only these two options, software wallet or hardware wallet and it is up to the user’s judgement to make the trade off.
Here, to illustrate, “browser compromised” means for example that your browser can swap out an URL under the hood and display a green padlock when there shouldn’t be one.
Now, in some further detail, II actually tries to improve over wallets like MetaMask. Where those wallets store key material permanently on disk (at least encrypted), II doesn’t. II only creates session keys that are valid for 30 min. The permanent keys are inside the biometric sensor or Yubikey. That is an improvement because it shortens the attack window for certain attacks.