Docs AI accuracy

Developers
1

I tested today how different AIs know the IC docs. I tested both a well-known AI and the ‘official’ Docs AI.

Background is that the well-known AI drove me crazy with a vetkd_ibe_decrypt_key method that doesn’t exist or doesn’t exist anymore.

So I started with some general test about how well the well-known AI knew the Rust CDK. It gave a detailed but pretty bad answer. I let the AI of the Docs evaluate that answer, and it confirmed that it was a good one, with only some minor issues. I then compared the answer with the ic-cdk crate myself (https://docs.rs/ic-cdk/latest/ic_cdk/bitcoin_canister/index.html) which revealed that the answer was not good at all. After feeding the index page into the well-known AI, I let the well-known AI come up with some questions that could have tricked the Docs AI. The question I picked was:

Does the ic-cdk crate expose a bitcoin_canister module?

The answer of the Docs AI was:

Based on the provided knowledge sources, the ic-cdk crate does not currently expose a bitcoin_canister module in its stable release. This is confirmed by a forum post where a user could not find such a module in the ic-cdk repository, and an official response indicated that this functionality was not yet included in the stable Rust CDK. Instead, developers are advised to invoke the Bitcoin canister directly for such functionality Forum Post: How to access the bitcoin block headers using the API?.
If you need to interact with Bitcoin-related APIs, you may need to construct the calls manually or use other available interfaces until the Rust CDK provides direct support.

Which is pretty wrong, considering:
https://docs.rs/ic-cdk/latest/ic_cdk/bitcoin_canister/index.html

Is the problem known and there is a way to keep the AI of the Docs updated with all the relevant docs? Because it is a pretty useful instrument.

I know that RTFM is the answer. But for all the lazy people around could not be possible to let also the AI of the Docs RTFM?

2

The Docs AI does RTFM regularly (maybe every 2 weeks?), but it is still an AI. I suppose in this case the problem is that for a long-ish time the way to talk to the BTC canister you were supposed to go through the management canister, so a fair amount of older content plus most of the forum posts (which it also reads) all point in a different direction.

Most likely if you pick something that has been around for a little longer in a stable state (0.18 of the CDK was finally stably released 3 weeks ago, the version before does not contain the bitcoin_canister module) where a little more content is avialable the results would be better.

TLDR: yes, we know it’s flawed, but we don’t have anything better at the moment…

1 Like