Ah, thanks for all of the info.
So we definitely will need to create a general-purpose CRUD database that automatically scales and allows complex querying, BigMap and BigSearch seem out for that then.
Now my question is, what is the best architecture to choose for this database? I know you’ve mentioned a lot about graph databases, I just want to make sure that’s the right choice. Please reason through this with me if you don’t mind.
Taking an existing relational database like SQLite or Postgres is probably out of the question for multiple reasons. SQLite would probably work well compiled to Wasm, but it will be stuck within one canister, so no automatic scaling. And as far as I understand, relational databases don’t do horizontal scaling too well (I could be wrong, I believe you can scale them horizontally, but I believe their underlying mathematical properties don’t lend to this well, and I don’t seem to ever hear about horizontally sharded Postgres or other SQL databases).
So if a traditional SQL/relational database is out of the question, which architecture is best? For vertical scaling, we’ve got about 4gb of storage and that’s it. As for processing power within a canister, I’m not sure what we’re dealing with. But anyway, we’ll need an architecture that can be sharded and scale out horizontally, extremely well.
So a graph database architecture seems like a good choice. But is it the best choice? What other architectures have you considered? What led you to the graph database?
This is going to be a foundational and indespensible project, I feel most apps are going to need a scalable database, as in it will be crucial for web-scale applications.
You mentioned creating an ORM. Personally, I’m more interested in just having the query language or the low-level API. It might make more sense to focus efforts there, and create a solid foundation first that ORMs and other projects can build off of. ORMs are notoriously bad, and getting them right I believe is a very difficult task.
To that end, that’s one of the major motivations for the project I’m about to start working on in a couple weeks. I’m not sure the exact name I’ll use yet, probably something like GraphIC, GraphICP, or ic-graphql. The idea is to create the ultimate GraphQL generator. You define a schema, and it will automatically generate resolvers for all basic CRUD operations, and eventually more. Essentially it will be an ORM, and IMO using GraphQL as your ORM is the best way to do an ORM that I’ve ever seen. The standardization and world of tooling will be unmatched compared to any custom ORM, especially one being built now from scratch.
But for my project to succeed and reach its full potential, it needs an infinitely scalable database API to implement the resolvers with. I was hoping BigMap/BigSearch were going to provide that implementation, but I guess not.
So my motivation here is creating the ultimate declarative backend, the simplest way to develop a CRUD application that has ever existed. I’m about to start dedicating most of my time to the GraphQL project, but I need this database.