If I don’t care what the throughput or efficiency of the datacenter is (few developers do); if I care only about my canister throughput; there doesn’t seem to be a reason to not use the number of update messages finalized per second per canister as a primary performance metric. Given a minimal real cost to update a single variable (literally one wasm instruction?) , the number of such updates successfully completed in a second (meaning you can then do a query and get the new data back) seem useful to me, measurable and consistent for any given set of hardware. The getting the data back is the kicker here - the upper limit is based on the number of messages updated with each block, the blockchain consensus overhead, plus all the normal network buffering and propagation delays (this can be truly significant). It seems like we don’t know the real numbers for the real hardware that is already being built?
The expectation of developers to build one app out of many canisters is a bit of a surprise. The questions I have raised about single canister performance are multiplied wrt multiple nested canisters updates and query calls too across multiple subnets.
The IC doesn’t do real transactions, does it? It processes messages but not transactions in the traditional sense as a set of updates all of which must succeed or fail together. The IC doesn’t make any such guarantees for more than 1 update message at a time? In fact, this is a bit of problem - transaction guarantees have to be provided elsewhere and the single message guarantees can’t be used in it’s place without something else to to it.
So a manager canister calls worker canisters. Each update call for a manager canister generates of a set of messages sent to other canisters. All of these messages must succeed and return consistent data while the manager processes it’s update message and finishes. If any one of them fails or times out or triggers a business logic error, then any update messages sent out to those containers must be rolled back somehow, otherwise the in-memory data structures will be inconsistent across the canisters. There is no provision in the IC to do that directly now, it must be coded by each developer of each canister(??). This isn’t better or worse than today - the complicated details of real life distributed systems make this more or less a problem you cannot solve generally; one adapts ad-hoc approaches to fulfill the underlying business requirements of the model, on top of real-life implementation deficiencies/complexities that always arise.
Given the design as I understand it now, I don’t see how splitting functionality across containers can improve the bottom line update performance of an app over one that is just in one canister. The performance of a manager canister will clearly be worse than the alternative if it makes any, even one, update call to another canister and then later continues to query that container or updates itself for bookkeeping. Each update to each other canister is resolved at some point in the future and only after that point will queries work. This means a multi-canister app cannot perform as well as single canister apps wrt updates or queries and the top performance basically will decrease linearly with every addtionally update or query call. If a single container can process 10 updates in a second, a container calling that one can only do 5/sec if it does one itself using the value returned from the first call via a query.
Beyond the standard difficulties designing and verifying any distributed system, the performance profile I postulate really makes the idea of decomposing an app into cooperative canisters unattractive.
Much of hype about queries is that they work on read only pages and so can be considered ‘fast’. This exaggerates the difficulty searching vs. just raw processing. If the query call executes a real algorithm and then loops through data structures constructing a result data set, that’s quite a bit of computation. If that query call itself must query another container, the unavoidable network overhead of inter-canister messages will decrease performance, I think substantially, because all network work is simply too slow. If the canisters are not in the same subnet I think one can forget about processing 100s of such read-only queries a second regardless; I believe ~10 transactions per second is about the best that can be done with the best code today over the public Internet if it involves sending two IP packets to two different destinations and waiting for results. Fundamental physical limits start mattering right away when one must communicate across the world. Canisters in two subnets on opposite sides of the Earth probably won’t be performant with 20K km of propagation delay between them.
The process of designing a cooperating set of actors that work properly while sharing no resources except source code is not an easy thing - in fact, it probably requires more skill and thought than single actor models. I argue that few developers today can confidently design or build such systems as a matter of course. It’s a domain visited by few, understood by fewer, and traditionally we, and our results, are very expensive. I’m also arguing that just adding canisters doesn’t clearly improve performance in this context in a way that’s meaningful to users or developers. Therefore, this idea that this will be a common thing in the IC needs much more exposition and information.
On the one hand, the IC doesn’t require you to write authentication, logging, databases, backups, twistd, etc. that cruft up lots of systems today. If, on the other hand, you need training in general system design, and distributed system design in particular, to create any significant system (say you need real transactions), well, that’s doesn’t seem like the IC is all that great.
Especially if, at the end of it all, you have a requirement for X updates/sec for one particular canister and it turns out X is bigger than what the IC will guarantee for any given canister. Then you simply can’t use the IC. One hopes one can decide one way or another before committing to the platform - hence knowing what the IC’s X is today, tomorrow, and in the next drop, is important.
If that X turns out to be less than ~10, I think the IC will not succeed as currently intended. Successful distributed systems often require a lot more (they can be too complex to grasp at once) as do most simple web-based apps that do form submits (lots of direct traffic) really. If a single canister/subnet cannot support 50 updates per second, and multiple canisters only reduce throughput relative to single-canister, then I cannot think of a single production internet application I’ve ever written that could be successfully translated and work in production on the real IC network and achieve identical performance.