@Maxfinity, thank you for your last-minute change proposal w.r.t. the textual representation. @roman-kashitsyn, @benji, and I just had a good discussion on the proposal and what has been discussed in the forum by @Maxfinity, @timo and others.
Here’s a summary of our findings.
The properties we want to achieve
A textual encoding of any non-reserved principal is a valid textual encoding of the default account of that principal on the ledger.
The decoding function is injective (i.e., different valid encodings correspond to different accounts). This property enables applications to use text representation as a key, for example in a map.
Protection against copy-paste errors or typos
Human readability (particularly the ability to identify the subaccount with the naked eye)
Approach
We concluded that the most suitable representation meeting those properties is the one presented in the following examples (note that the principal contains a checksum over itself):
4kydj-ryaaa-aaaag-qaf7a-cai:1 (simple subaccount, no checksum on subaccount)
4kydj-ryaaa-aaaag-qaf7a-cai:3fCe35D21Aa8 (complex subaccount, contains checksum over the whole 2-tuple through the case of letters in the hexadecimal representation of the subaccount)
Informal specification
Let f be the textual encoding function specified as follows, where || is string concatenation. Let principal be a principal in textual representation and subaccount a subaccount in byte array representation.
chk(a, b) is a checksum function that capitalises b based on the SHA-256 hash of a. The input string a is a canonicalized hexadecimal string, i.e., comprising only the characters [0…9, a…f]. The hexadecimal representation a is hashed with SHA-256 to obtain h, and for each digit with index i in a, print it in uppercase in the result, if the 4*i-th bit of the hash is 1, in lowercase otherwise. Digits are taken over from a to the result. I.e., we capitalise letter symbols of b in the output based on the hash h of a. This is analogous to Ethereum address checksums.
In words
The encoding is created from the principal, followed by a colon, followed by the subaccount
The subaccount is defined as follows:
Take the subaccount in hexadecimal representation with leading zeroes stripped
Compute a checksum over the principal and subaccount and represent it through the case of the letters in the hexadecimal representation of the subaccount (the principal remains untouched)
Question / discussions
Do we need to include the “:” in the input to the checksumming, as a “domain separator”? Does not harm, but not clear whether it is really needed. If not, it should be removed.
Why does Ethereum EIP-55 use the 4i* and not just i? For a good hash function, this should not make any difference.
Should we rather do the hashing on the byte-array representations? Might be cleaner, it would be an easy change that we need to discuss.
What does this achieve?
This checksum is part of the resulting subaccount only if the hexadecimal representation of the subaccount contains letters. I.e., for “simple” subaccounts like 1, 2, etc. there is no checksum available for this reason as digits don’t have a case. For subaccounts derived through a hash function, e.g., SHA-224 or SHA-256, we have an expected ~21 or ~24 bits of checksum, respectively, expressed through casing of the letters, which catches copy-paste errors with high probability. Simple accounts like “1”, “1234” etc. do not have a checksum.
We leave the principal untouched, so it can be easily compared via eyeballing.
We have a checksum over everything if the subaccount is not a “simple” subaccount. Having checksums over complex, long subaccounts addresses requirements addressed in the forum threat. Not having checksums over short, simple subaccounts seems OK in the light of the discussions.
Having upper/lowercase in the subaccount has not been seen as an issue so far (but also not explicitly addressed). @timo?
Users can still create simple subaccounts on their own as they are not checksummed.
We think that this approach is the best compromise we can make given the discussion we have had so far in the forum. It has checksums where helpful, but skips them where we think that they are less required. We think this is a clear improvement over the previous proposal.
Please let us know what you think about going forward with this proposal. If we do not hear objections, someone needs to spec it properly and then we can open a vote on it. At least it seems that what we have now is strictly better than what we had before.
Is the checksum optional? That is, can a user opt-out and just write everything lower case if he wishes and forego the benefits of the checksum?
Do we need the checksum to include the principal? Isn’t it sufficient if it is computed from the subaccount alone?
The principal’s internal checksum is based on CRC32. Instead of introducing a new function, sha256, would it make sense to use crc32 again to reduce code dependencies and maybe it is also faster?
We have no checksum for small subaccount ids and >20bit for large ones. But what about the middle size? I am sure there are application where subaccount ids are generated sequentially and handed out.
For a 4 byte size (8 characters) we have on average 3 bits of checksum. For a 8 byte size (16 characters) we have on average 6 bits of checksum. Compare that to an IBAN which has ~6.5 bits.
If we consider that number of bits fine then I think we are better off adding a fixed length of 1 byte (2 characters) to the subaccount id, i.e. instead of
It avoids confusion for users who aren’t used to the capitalization. Moreover, with a 4 byte subaccount 1 out of 4 subaccount ids will be all lower case or all upper case. Then the user will wonder if what he is looking at is checksummed or not. These things are avoided by adding dedicated checksum characters.
Hi @dieter.sommer, @Maxfinity,
This would remove the copy-safe characteristic of the current textual representation.
In both of these examples it is very easy for a user to only copy the principal and miss the subaccount. One of the main benefits of the current textual representation is that it is 100% copy-safe. A user cannot mis-copy any part of the account-id because no part is valid without the whole. This is a very valuable feature.
I think in most cases a dao will not help a user who sends tokens to the main account when the tokens are supposed to go to a subaccount, the tokens may be already transferred out of the main account by then or similar.
It is also straightforward and easy for a dapp to have a UI where the user can create subaccounts using simple numbers (1,2,3) and the dapp-ui will generate the account-id for it.
It seems that the following argument of @levi applies here:
Our impression of earlier discussions was that we could forego the checksum for “simple” account ids, striking a balance between simplicity and checksumming. But it seems there are strong opinions of having everything checksummed, always, and an explicit checksum somewhere in the string. It seems there are multiple strong opinions to go for the <<principal>>:<<subaccount>>:<<checksum>>
approach following the latest discussion. I.e., one of the approaches @timo has outlined above.
Thanks for the active discussions and bringing this forward.
Other opinions on the importance of the whole account id being checksummed, always?
I think that the whole_account = <>::<> being checksummed always makes sense.
Consider the use-case of invoices for a customer, where a customer is a principal.
customer1::1 would be simple account_id
customer2::12345 would also be valid
customer3:: “long description of account id” → hashed to something smaller
I think @levi 's argument was different. It was primarily about the principal alone being a valid account encoding, hence it would be possible to miss the subaccount when copying and still end up with a valid account. That argument has no bearing on whether the checksum should be taken over the subaccount alone or the principal-subaccount combination. Because in the scenario that @levi is taking about the user has missed copying the checksum, so the nature of the checksum cannot have an influence on the outcome.
Several design trade-offs here:
i) Ease of reading vs explicit checksum
Embedding the checksum as capitalisation is easier to read than having an additional checksum as prefix or postfix. It’s clear what the actual principal and account IDs are, without needing mentally parse/understand the seemingly random checksum byte.
ii) Ease of subaccount generation manually, without using an external tool
In the capitalisation checksum, for degenerate cases (subaccounts 0-9) there’s no need to calculate the checksum, which is convenient because most such use cases would be transferring manually between subaccounts.
Decisions we need to make:
How to represent subaccount 0? We have previously decided that subaccount 0 should be represented as just the principal. Why was it decided that way? Wouldn’t it make more sense to represent it as principal:0?
CRC32 vs SHA256
Since we’re never encrypting the principal:subaccount with a stream cipher I see no problem with CRC32.
@roman-kashitsyn proposed that we should use Crockford’s base32 encoding for the subaccount instead of hexadecimal for a more uniform representation, with the additional benefit of increasing the number of bits of checksum for large subaccounts.
Shall we use explicit checksum bytes?
Based on my reasoning above I would tend towards no.
Is checksum optional?
No. Because we don’t have backwards compatibility issues like ETH and a single unique representation is always preferred.
To discuss some details about this approach. What do you think about these?
Place the checksum in the middle so that the subaccount is more easily visible to the user at the end. Just in case there are scenarios where it is important to see, e.g., that the subaccount id matches a hash or is a certain small number.
Choose - as the separator. That seems easy on the eyes because the dash is already present on the left. So it would look like this: 4kydj-ryaaa-aaaag-qaf7a-cai:1b-3fce35d21aa8. It puts a little less emphasis on the checksum, not creating the impression that there is a third material value. And finally, it makes it easier to make the checksum optional (see next point).
Make the checksum optional. For example both 4kydj-ryaaa-aaaag-qaf7a-cai:1 and 4kydj-ryaaa-aaaag-qaf7a-cai:1b-1 are valid for subaccount 1.
In my opinion the decision should be between
a) the original proposal that was already spec’ed and that does not visibly expose the subaccount id, and
b) the proposal that we are discussion now, started by @Maxfinity, but with an optional checksum
A checksum is only needed when there is transmission or copy-pasting going on. If we hand-craft an account id from the two components then a checksum does not help against errors (the errors would be made already on the inputs of the checksum). Instead it is a nuisance because it requires a tool to compute the checksum. If an account id is software generated and then passed to another software then a checksum does not help. Again, it is a nuisance because another library dependency is needed. Therefore, I think for scripting and command line use there is a huge benefit of leaving the checksum optional. We can add the following as property (5) to the list that @Maxfinity made:
(5) an account encoding can be hand-crafted from its two components.
Then proposal a) has properties (1)-(3) and b) has (2)-(5).
I think we should take (5) into account when deciding between a) and something else. And if we go with something other than a) then not leave (5) on the table. Or, by contraposition, if we need a tool to manufacture an encoding, i.e. if we don’t have (5), then why not stick with proposal a)?
(I have no personal preference between a) and b). Both look good to me.)
Then this proposal satisfies all properties (1)-(5).
Namely: <<principal>>:<<subaccount in base32 with capitalisation checksum>>,
where the capitalisation checksum is a bitmask of CRC32(<<principal>>:<<subaccount in base32>>).
It seems impossible to have (5) with a capitalisation checksum. For (5) the checksum must be optional. And a capitalisation checksum cannot be optional because it naturally occurs with a certain frequency that all letters are lower-case in the checksummed version, in which case the receiver cannot tell if the sender excepts the checksum to be checked or not.
The property (5) is achieved with the assumption that hand-crafted subaccounts have low IDs (0-9), and machine-generated cases have high subaccount IDs. Thus handcrafting 4kydj-ryaaa-aaaag-qaf7a-cai:4 is arbitrary.
Moreover the checksum is not optional, thus all lowercase/uppercase must be valid checksums too.
Would you basically round off the preceding As? This library outputs uppercase…I’m guessing adding the CRC32 checksum would be something we’d need to add somewhere.
Why does [1] Encode to AE? Shouldn’t it just be B? I may be missing something significant about Base32.
What I need: a motoko library that converts a subaccount [Nat8] (usually of length 32) to the proper account part of the textual representation.