IC restrictions and aspirations versus general-purpose computers

[pie-silo]

(Splitting this out from https://forum.dfinity.org/t/time-and-deterministic-code/1396/7 at Enzo’s request.)

Dominic has previously written that:

Essentially, the Internet Computer supplies an end-to-end solution. You don’t need anything else, and we try to squeeze out every vulnerability. To us it doesn’t make any sense having dapp developers using AWS hosting trusted websites to provide an interface to DeFi contracts on Ethereum, say. Plus, we want people to completely discard legacy IT systems such as filesystems, databases, middleware, Web servers, cloud services, CDNs, firewalls. You just need the Internet Computer.
…
The purpose of the Internet Computer is to replace legacy IT…

But then Enzo says

I don’t know of anyone making claims about replacing “all existing computing infrastructure”.

Enzo goes on:

The internet computer is a general purpose computing platform with powerful tools for abstraction, and yes, for a vast number of use cases, you don’t need anything else. Like most complex pieces of software, it makes trade-offs, and optimizes for some things at the expense of others. Unfortunately, there is no access to a high resolution clock.

Now if we were talking about niche capabilities like HSMs or FPGAs, fair enough. Even fairly mainstream capabilities like, say, SIMD, ML accelerators, GPGPUs, 64-bit memory spaces, ok, they’re widely used but many apps don’t need them.

The point where I find this a bit weird is the restriction to only deterministic functions: no random numbers, no accurate real-time clock, no networking or external side effects, no internal concurrency, no scratch storage, maybe no private data, … Those are pretty basic and well-established things for general-purpose computers, and have been for decades. Games are mentioned as a key use case for the IC but almost every game uses random numbers.

I think every app of any size that I worked on used needed at least one of these. Even in Haskell or Erlang practical complete systems are not always 100% pure, deterministic, and side-effect free.

I recognize that sometimes giving up X gives you benefits of performance or simplicity somewhere else but these seem like a lot to give up…

[lastmjs]

I echo your concerns, DFINITY is reaching far but obviously there have to be many limitations for years to come. Are we correct with our concerns? Why do high-profile team members have conflicting views here? This is one of the red flags that makes me uneasy, I’m afraid some of the expectations being projected are unrealistic.

[Steve]

You point out some important limitations. A few of these have some not-so-painful workarounds I’ve come up with / found:

For random numbers you can use some deterministic random bit generation algorithm and allow a trusted principle to periodically update the keyset. Naturally, this is a bit delicate. I have also found this, but not sure what the plans are really.

Networking is a bit tricky, but you could set up some trusted oracle that polls the IC for data requests, and republishes into actor - not ideal, but will work for a lot of situations.

AFAIK - the memory page is private to the actor, and plans exists to create a private subnet with an added layer of security encryption that memory page.

No Internal concurrency is a bummer. I’m not entirely sure on all the details, but I believe, provided your states are isolated, you could publish update messages to multiple actors. I’ve seen some slides in the past that better explain the process but I can’t find them at the moment.

I think we can all agree that Dfinity is still in its very early days of development and that Dfinity’s end dream is a very ambitious one - to say the least! Even ignoring the above workarounds, you can still build a massive amount of applications that solve real world problems. Is Dfinity the best tool for the job in its current state? Probably not, but its a viable one. Will the “finished” version of the IC have some of the same above limitations, maybe who knows.

Personally, I’ve enjoyed the developer experience so far, and I really hope the team keeps pushing on this. Even if Dfinity never grows to the point where it can truly replace the entire IT stack or become truly unstoppable I would still be satisfied if they launched as a “Heroku” that embraced the current development flow.

[wang]

A Random module was merged recently, so it should be available in the next release.

[enzo]

I was absolutely referring to capabilities such as ML accelerators, which I stated explicitly in my post. With great respect for the author here, I find this selectively edited narrative highly disingenuous.

Addressing the latter points…

@pie-silo

deterministic functions

We traded a non-deterministic execution environment for stronger security guarantees.

@pie-silo

random numbers

Randomness is available, but needs better documentation.

@pie-silo

networking

We traded network interfaces for the simplicity and convenience of canister interfaces.

@pie-silo

external side effects

This is a symptom of determinism. There are workarounds using oracles.

@pie-silo

internal concurrency

We have plans to introduce internal concurrency.

@pie-silo

scratch storage

We use orthogonal persistence so your scratch storage is the heap.

@pie-silo

private data

Please encrypt offline or use homomorphic encryption until we have sub-networks with SGX support.

[Ori]

(@Steve Aside if you’re interested, the random-beacon repo you found is part of the underlying consensus layer, it’s used to choose which nodes’ blocks to commit, amongst other things.
You can use the Random module Norton posted above for accessing randomness in your projects.)

[pie-silo]

Hi Enzo, I’m not trying to misrepresent your position.

Please take on board the feedback that different people from Dfinity appear, to an interested outsider, to be making contradictory claims. I’m just trying to elucidate what the value proposition actually is.

I think you are saying: some very basic and commonly used functions of general computers are admittedly not available. Dfinity’s proposition is that almost all systems don’t actually need these capabilities, even though they use them today. Further you believe that the IC will have advantages of cost/simplicity/performance/security that will motivate developers to move to system without these features. Is that a fair and accurate summary?

If that’s it then it sounds like a classic Innovator’s Dilemma argument, and it does sometimes work: people give up features previously seen as essential to get something cheaper or otherwise better. But it does hinge on the customer value of the specific tradeoffs.

My original question was not about exotic hardware like ML accelerators, but about accurate real-time clocks, obviously a very basic thing that has been on even tiny computers for decades, and used by almost all applications.

On the specifics:

@enzo

We traded network interfaces for the simplicity and convenience of canister interfaces.

I understand you made that tradeoff. I wonder if it is realistic to expect to migrate all (or almost all) IT systems to a platform that can’t open network connections. It seems like a pretty basic feature in 2020.

For example if you look at AWS Lambda, an arguably analogous serverless platform, functions can make outgoing network requests in handling a request.

@enzo

Randomness is available, but needs better documentation.

Great, looking forward to it. I sympathize with having an under-development system with partial documentation.

@enzo

[lack of external side effects] is a symptom of determinism. There are workarounds using oracles.

Oracles are essentially a way of moving the computation that can’t be done on this platform into another more-capable platform. There’s nothing wrong with the technique, except that Dominic repeatedly asserts that the IC will replace all (or maybe nearly all) other systems and people won’t need to manage other infrastructure. He specifically criticizes Ethereum for needing external helpers.

So there’s an apparent inconsistency here between Dfinity on the one hand promising to replace (almost) all IT systems, and on the other hand saying developers need to run external helpers somewhere else to do pretty basic features.

@enzo

We traded a non-deterministic execution environment for stronger security guarantees.

That’s probably a whole different thread but it would be great to see what specific security guarantees Dfinity makes to app developers.

@enzo

Please encrypt offline or use homomorphic encryption until we have sub-networks with SGX support.

So saying “encrypt offline” seems to mean, “run part of your system outside of the IC” and again, is really in conflict with assertions that people don’t need other infrastructure.

Homomorphic encryption of all the data, maybe can work, but that sounds super hard for app developers to do safely within their app, without support from a compiler or framework.

[PaulLiu]

Just want to emphasize that canister computation is not “side-effect free”, but rather they are “effectful”. The effects come in two flavors: mutation of its own state, and sending messages to other canisters.

Message based communication is a higher level of abstraction than network connections. They may not address all problems, but their use case do go very far. IC’s messaging in particular provides a guaranteed reply in terms of either a success or a failure, so a lot of complexity is already taken care of by the platform.

The most obvious limitation is communication with legacy IT stack. There are various workarounds to this, in similar ways to how other blockchains integrate with existing IT stacks.

Personally I view it as a challenge to think differently on how existing use case can be better satisfied without resorting to legacy stack, when messaging is reliable, persistence comes as default, and computation is trusted.

[pie-silo]

@PaulLiu

Just want to emphasize that canister computation is not “side-effect free”, but rather they are “effectful”. The effects come in two flavors: mutation of its own state, and sending messages to other canisters.

Hey PaulLiu, thanks for clarifying that. You’re right, my choice of words was imprecise.

This raises a question, though: can read requests send subsidiary mutating requests to other canisters? It seems like they can. Isn’t that a problem if the read requests are run with typically no validation that the computation is correct, but they can indirectly mutate state?

@PaulLiu

Personally I view it as a challenge to think differently on how existing use case can be better satisfied without resorting to legacy stack, when messaging is reliable, persistence comes as default, and computation is trusted.

I also find it a really interesting challenge, and I like the elegance of the CSP paradigm in general. It’s great that people are trying something novel. That is why I’m asking questions.

Quite possibly some things that seem like major blockers now will turn out not to be important. I try to keep an open mind on that.

But there seem to be very significant blockers to wide adoption that are as yet unanswered. It’s easy to say “computation is trusted” or “it’s unhackable” but the technical information published to date is not sufficient to establish why it should be trusted, and in fact raises more questions than it answers.

[Ori]

@pie-silo

can read requests send subsidiary mutating requests to other canisters?

Read requests/query calls can’t make update calls to other canisters (they can’t actually make any calls to other canisters at the moment, but that will change), there would need to be some validation for this.

[enzo]

Apologies for this delayed reply. I am still out of the office through Sunday.

@pie-silo Your feedback is valuable to myself, my colleagues, and the organization, and I encourage you to seek clarification wherever possible.

The Internet Computer uses a WebAssembly-based virtual machine in its primary execution environment. The virtual machine is provided a set of host functions that extend its capabilities beyond WebAssembly. WebAssembly by itself consists of mostly primitive operations on 32- and 64-bit integers and floats. The set of host functions give rise to the canister abstraction. As we extend the set of host functions over time, canisters will see more capabilities available to them, like the ability to perform computation on encrypted data.

I believe the Internet Computer is an end-to-end solution for a wide variety of use cases, that the number of use cases will grow with its capabilities, and that canisters offer a compelling value proposition for customers who enjoy simplicity and can tolerate some added latency for enhanced security.

The accuracy of clocks is affected by system latency. We should be careful not to compare apples with oranges here though. Our clock actually returns the time of day that was mutually agreed upon by multiple data centers who don’t necessarily trust one another.

Regarding network connections, and as @PaulLiu mentioned, message based communication is a higher level of abstraction. If customers value abstraction, and I believe they do, then they will find the Internet computer a compelling platform.

Probably best to start a different thread on why we require deterministic computation and its implications for system design and user experience. I don’t dispute the implications are significant, but many perceived handicaps turn out to be quite manageable. I stand by the design decisions we made here to optimize for security and the best possible user experience.

You mention Ethereum. Ethereum does not provide access to a clock, unbiased randomness, horizontal scaling, or many other basic features. Ethereum relies on external helpers in many places where the Internet Computer does not.