I am also looking for a “formula” for what the correct behaviour in case of failed operations is, but couldn’t find one yet. (See my related question Is trapping better than returning an error?)
I think you are free to choose between trap and reject, depending on whether you want to roll back or not. The handling on the caller side is essentially the same. The caller sees an error code and an error message (Text). I don’t think trap is reserved for certain situations.
That said, even in error cases it could be that you do not want to roll back, even if you normally would. For example, a canister might want to track who (which caller) made the failed call because the canister might still want to charge a fee for the call.