[Blog post] IC internals: orthogonal persistence

TL;DR IC internals: orthogonal persistence

Orthogonal persistence was the first feature I worked on at DFINITY. I still have warm feelings for that feature; I would love more people to understand it deeper. The article is not beginner-friendly; it targets experienced developers who want to dig deeper into the tech.

Would you like to see a similar technical article on some other aspect of the IC? Answer in a comment.

Big thanks to @ulan for reviewing the draft of the article.


Very interesting read, would love to see more.

Do you already have an idea how big files will be stored? Will they always be loaded in memory?

I assume you mean checkpoint files. Checkpoint files are memory-mapped, which allows the kernel to be clever about loading pages into RAM. Only actively used pages need to occupy physical memory, and the kernel can always reclaim those pages because there is a copy on disk. This approach scales to terabyte-sized states.


@roman-kashitsyn Fantastic blog. I especially enjoyed your post on replicated state machines. Do you still plan to do a post/posts on the topics of “how different subnets communicate with one another, how individual worker bees form the swarm”?


as of April 2022, 95% of message executions change at most seven memory pages.

Uh, real numbers, exciting!

I wonder how many of them only have 7 non-static memory pages to begin with. But that’d be slightly harder to measure.

Such memories open up attractive optimization opportunities, but we did not have enough pressure to implement them.

Sounds like you are sad that pressure wasn’t high ehough :slight_smile:

1 Like

Thanks @roman-kashitsyn , always a pleasure to read your blog posts.

This one was super interesting as well, thanks for sharing, it is much appreciated !

Andreas also proposed short-term memories with a lifetime of a single message execution. Such memories open up attractive optimization opportunities, but we did not have enough pressure to implement them.

Could you elaborate on this one please ? What kind of optimization you have in mind ? What kind of pressure would have been useful in this case ?


One of the benefits of short-term memory is that the runtime doesn’t need to track dirty pages, so this memory might be measurably faster to use.
The Motoko runtime could use the short-term memory as fast minor heap. Rust canisters could use it as a scratchpad with a fast bump memory allocator.

If the Motoko team was struggling with improving their GC algorithm and short-term memories was the easiest way forward, we could have implemented this feature. Currently, there are plenty of other low-hanging fruit.

Yes, I like the idea a lot. However, it’s definitely not the most important feature to work on right now.

1 Like

Great article, thanks for sharing!

A very basic question: what happens if a replica crashes (or node goes down) in between checkpoints? If the in-memory running page deltas are lost, how does the replica recover the actor memory snapshot after it restarts?

There are two recovery scenarios:

  • If the replica was out for a brief period of time, the faulty replica might be able to catch up by replaying recent blocks on top of a checkpoint. The execution is deterministic, so the replica should arrive at the same state. Consensus keeps state machine checkpoints in sync with its block storage.
  • If the replica was out for a long period of time, it might need to fetch a new checkpoint from peers. State transfer protocol covers this scenario.

Nice! Another article to add to my IC reading list.