I’m reaching out to discuss an optimization proposal aimed at improving the efficiency of IC nodes through the concept of segmented state persistence. This approach has the potential to significantly reduce the memory demands on nodes, thus enabling more complex and numerous queries to be executed concurrently.
As we currently understand, when a query is executed, the state that the query operates on must be retained until the query’s execution concludes. The prevailing implementation tends to hold on to the full replicated state of the subnet, which can occupy tens of GBs of extra memory, even for regular queries that require only the state of a specific canister. This information was detailed in a discussion found here: Deterministic Time Slicing.
The implications of this are particularly restrictive for AI applications and potentially other areas where queries could benefit from a larger capacity. Currently, to safeguard the subnet’s liveness, the size of queries is artificially limited. However, allowing for larger queries and a greater number of them to run concurrently could significantly expand the IC’s functionality and efficiency.
The proposed segmented state persistence would involve retaining only the necessary parts of the state relevant to the queries being executed, rather than the entire replicated state. This method poses a particular challenge for composite queries, where the involved canisters might not be known beforehand, but I believe a strategic approach to predictively segmenting or dynamically managing state could offer a solution.
Enabling each node to process more queries through this optimization could serve as a substantial leap forward for the IC, accommodating the growing needs of complex applications without compromising the network’s integrity or performance.
I welcome thoughts, feedback, and any discussions on how we can collaboratively work towards implementing this proposal or exploring alternative solutions that address the same underlying efficiency challenges.
Thank you for considering this proposal, and I look forward to our productive discussions.
The reason why holding on to the full state while executing a query is a potential issue is that if said query runs for too long, the replica has to hold on to the full subnet state for that long.
This would only mean “holding on to the deltas / changes between that state and the current state”, so by no means a full in-memory copy of the state. But, if the state was “old enough” it could easily add up to tens of GB.
However, having to hold on to an old state is not the only reason why limits are imposed on query instructions / durations: there is only a limited number of available “query threads” (because these are, in fact, separate “canister sandbox” processes that are quite a bit heavier than your average async task / light thread running inside a Web2 service). I believe (and I haven’t checked this with the Runtime team, so take this with a grain of salt) that the number of “query threads” could conceivably be increased a bit, particularly if they were executing long running queries, instead of constantly switching among canisters. But given that the number is currently (I believe) 4, there’s only so many “long queries” one could run concurrently before all queries submitted to the subnet would start accumulating in a backlog, resulting in huge query latencies.
In other words, making it possible to only retain the state of (one or more) specific canisters during query execution is a necessary but not sufficient prerequisite for meaningfully increasing query throughput on a replica.
As a side-note, it was observed recently that certified queries actually require a fixed proportion of a subnet’s nodes to execute. Meaning that while regular queries scale linearly with the number of replicas (the more replicas on a subnet, the more queries the subnet can handle); certified queries won’t (assuming that you can run 4 concurrent queries on a replica; and that you need 1/3 of a subnet’s replica to run a certified query; you can run a maximum of 12 certified queries on a subnet, regardless of size). What I;m getting at is that any optimizations in query throughput may be needed to make up for the higher load imposed by certified queries, rather than aimed e.g. at increasing query instruction limits.