Does rust-connect-py-ai-to-ic have a tokenizer?

Developers
1

Hi! @jeshli , it’s me again, do you have a rust-connect-py-ai-to-ic tokenizer? Otherwise the input and output are tensors, not readable text :blush:

2

Yes. It’s in one of the branches.

3

Oh I found it, but I noticed that the return value of tokenize_text is vec int32 while the parameter of model_inference is vec int64? If these two datatypes don’t match then how do I use tokenizer_backend please? Happy New Year btw :blush:

4

Should be able to into() the type.

But for reference, we just open-sourced new resources for implementing models and tokenizers in a more straightforward and cleaner way.

It is actively being extended to work directly with HuggingFace’s transformers library which we are actively contributing to. Making it easier for anyone to deploy any model to the Internet Computer.

5

Wow, amazing project! :slight_smile: I’d like to ask you about a Transformer-based Time Series Prediction model that we trained and tried to upload to the project backend, but setup_model reported an error, can’t this project run the Transformer model?

Error: Failed while trying to deploy canisters.
Caused by: Failed while trying to install all canisters.
Caused by: Failed to install wasm module to canister ‘predict_demo_backend’.
Caused by: Failed during wasm installation call
Caused by: The replica returned a rejection error: reject code CanisterError, reject message Error from Canister bkyz2-fmaaa-aaaaa-qaaaq-cai. Canister called ic0.trap with message: Panicked at ‘not yet implemented’, /home/ydr/.cargo/git/checkouts/tract-01a3138c7a4b58a3/2a2914a/ core/src/ops/change_axes.rs:856:17.
Consider gracefully handling failures from this canister or altering the canister to handle exceptions. See documentation: http://internetcomputer .org/docs/current/references/execution-errors#trapped-explicitly, error code None

6

I would be guessing but perhaps Tract-Onnx does not have that function implemented. Have you confirmed that you can run your model as Tract-Onnx outside of the canister?

Candle also has Onnx support. I suspect that it has more active development. I can research making a cleaner version of what you are using py-connect-ai-to-ic using Candle instead of Tract but it will take some time.

Other than that I would re-write your time series model in Rust.

7

I created a rust project locally as you suggested and tried to run the model as tract-onnx, but it reported the same error. :sob: I’ll try Candle

8

I briefly tried Candle, but it usually uses Device:cpu or gpu when loading models in it, is Candle supported in canister?

9

The repo that I showed has an example which deploys a gpt2 model running with candle.

Yes you need to use Device:cpu, as you would see in the example code, because the IC does not have GPUs … yet.

You would need to be able to either write models in Rust or wait for me to develop (or develop yourself) a wrapper Candle-Onnx integration that works the way that rust-connect-py-ai-to-ic does.

2 Likes
10

Oh I found it! I will study this code carefully, thank you so much for so many days of patient answers and replies :innocent: , looking forward to your new project! :blush:

1 Like