Orthogonal Persistence vs Legacy Schema

Hello there.

As a developer, you would have a persistence at the (very) backend and create some efficient persistent mechanism for your data model (such as Schema, tables, relationship between tables, or NoSql, etc).
I learned that Dfinity introduced what they call Orthogonal Persistence (fancy name btw).
I wonder how can i understand this mechanism? I heard that the canister brings along your app as if every page contained the data itself. But is it a efficient model to store data like this, especially to scale up your end users (clients)? And can you manage the data or the protocol ‘takes care’ of the data model? How can we deploy a statistical model onto this kind of persistence?
Any more technical explanation of how this Orthogonal Persistence actually works as opposed to traditional way of persisting data?

Maybe too many question :sweat_smile:

Cheers,
Rich

6 Likes

Hey Rich, Orthogonal persistence is neither new nor specific to DFINITY. The Internet Computer platform from DFINITY persists the process memory of the WebAssembly programs that run on it. WebAssembly has a very simple linear memory model. When a function exported from WebAssembly program is invoked, and if that invocation modifies memory, then at the end of execution, the platform takes a snapshot of the memory. The snapshot is derived from a transaction log of modified memory pages. The platform uses the snapshot to instantiate the WebAssembly program when the next invocation occurs. Warmest regards, Enzo

6 Likes

If you’ve run virtual machines with Virtualbox or Hyper-V on your local computer before you may have seen them spend some time “syncing” or “persisting data” after you shut down the VM, this is what they are doing. They also take snapshots/checkpoints of the state this way while they are running.

5 Likes

An older repo, but essentially this: https://github.com/dfinity-side-projects/wasm-persist/blob/master/README.md

2 Likes

Hi! I have a similar, general question. In the old days I might write some java objects for behaviour and a JSP for a user to see data and do things by clicking around. I would normally leverage a Database like Oracle or postGres to store data that my application uses, using CRUD ops.

If I migrate my old app to DFINITY, how would I recreate my basic database functionality? Do I basically have to build this database myself for now by hand using your objects and persistence abstractions?

Thanks…

1 Like

Hello there.

In practice orthogonal persistence means “just write code”. You don’t need to build a database at all.
Let’s suppose, you want to store usernames of your users. In a service code you just declare some kind of variable (pseudocode)
let usernames = new List<Text>();
and then each time you want to persist a new username you just
usernames.add(newUsername);
That’s all.

2 Likes

Got it, thanks. This is sort of the answer I was expecting. Will try this approach…

2 Likes

If i can add some more, i think to see if from the glasses of classical data model, you would view the data differently, like Senior.joinu mentioned.
I don’t know exactly how you would make statistical model (would you extract the linear model like Enzo mentioned above and create your own data pipeline? )

Of course, this is an exciting platform, especially because we are to see a different data model entirely.

2 Likes

Related question, how does Orthogonal persistence work with large data sets.

  • Memory management? (Is there some kind of paging / swizzling ?)
  • Search and Indexing? (SQL is annoying but it does allow fairly sophisticated performant queries, are there libraries/datastructures for efficient search)
2 Likes

I would like to understand if all data in Orthogonal Persistence is public… I guess yes… while in Legacy DB there is always at least an admin account.

2 Likes

You always can encrypt your data to keep it private :wink: