@simdi.jinkins Thanks for correcting my misunderstanding of how and when re-balancing happens (when node “full” threshold is hit) and taking the time to answer my questions.
I hope you don’t mind me asking few follow-ups 
(Follow up Q1)
I’m still not quite clear on the details of how when a new canister is spun up (we increase # of nodes from N → N + 1 → N + 2), how the client becomes aware not just of how many canister nodes exist and the canister node partition to send the request to, but the specific canisterId corresponding to that node.
When the management canister creates a new canister parition node, it generates a completely random canisterId along with that node, one where the application developer cannot fix, control, or predict what the canisterId will be (I wish we could do this!).
How would a client then be able to guess this random canisterId when making requests if it has no control over the canisterId generation? There must be some “fixed” initial canisterId representing Canister A in the client application that the client can first use to make contact with the data storage solution, correct?
I see this as your solution to the issue I mentioned above. The client is told a single canister that they will interact with, and they use this initial canister to then get all information about the current array of canister ids and the current hash ring that they can then use AnchorHash algorithm with on the client side. If a re-balancing happens during the middle of a client session, the client needs to re-fetch the new array canister ids and the new hash ring.
However, this approach means that all clients are still originally directed to the same initial canister for fetching this cluster data, and that each application client will have a hardcoded canisterId that they will use, since they can’t predict the canisterIds of dynamically created canister nodes.
Is this a correct assumption?
(Follow-up Q2)
About a month ago, I had these exact same questions about data transfer speeds (inter-canister, ingress, egress, etc.), and the question got pretty much ghosted and went unanswered 
Whether no one has been brave enough to test this out or just hasn’t made their findings public, there are multiple anecdotal comments from developers who have mentioned that downloading 1-2 MB of data from the IC can take a few seconds in this thread.
By the way, consistent hashing is pretty cool - I didn’t know much about it before reading your solution → this article has a pretty neat breakdown with visualizations on how one might naively implement consistent hashing and how it works in general.
Looking into consistent hashing, it looks like for each new canister node being spun up the rebalancing cost is O(K/N), where K = the number of keys and N = the number of nodes.
This would mean that as you scale up, for each new node added the performance cost of data transfer redistribution remains constant.
One additional question (new)
- Thinking about consistent hashing and the way that it further subdivides the hash ring as it scales, how does consistent hashing handle unbalanced data coming in, or more specifically how does the AnchorHash algorithm handle it? Let’s say I’m seeding my scalable storage database and all of my keys are inserted in monotonically increasing order. How would that affect this architecture?