So this (prompting of the yubikey pin) happened for me as well. It is now no longer happening for me as well.
I am trying to see what changed and how. This led me to try to do a reproducible build for internet identity as described here (Verifying the code of the Internet Identity service (a walk-through)).
I have been yet to get to a current reproducible build. The build currently in production diverges. That said, this is a recent build (October time frame).