Hey guys, I’m running rust-connect-py-ai-to-ic’s GPT2 example and I’m getting an error when I run GPT_kv.py. Anybody can tell me how to fix it? Thanks a lot! Here are error messages below: onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Invalid rank for input: past_key_values_input Got: 6 Expected: 5 Please fix either the inputs/outputs or the model.
1 Like
Hey @jeshli Would you be able to help?
1 Like
Oh! @jennifertran, haven’t seen you for a while since the last time I posted! I’ve been plagued by this problem these days. Thanks for @ing~
1 Like
I’ve been away on vacation and then was busy on a few other focus areas!
Anyways, Jeshli can help! Also, if you have a link to the public repo, I can take a look as well. It seems that the original rust-connect-py-ai-to-ic project link is no longer open-sourced.
1 Like
Hey, @jeshli , I pulled down the code repository and followed the instructions to install the environment, but I keep getting errors when I run to the script GPT_kv.py, it seems that there is a problem with the input/output model, one dimension is missing. I’ll appreciate it if you could give me some help!~
Hi. I’m not entirely sure what is the cause of the error. Someone else had an issue from converting op-set 11 to 14 in order to make it work but then it broke in tract-onnx. You can review this thread Discord in which I provide a link to the onnx model. That will resolve the issue you are facing.
2 Likes
Thanks, @jeshli , I just went to the Discord and download the onnx file, but the out come was the same
@jeshli , here are the full error messages: torch.Size([1, 4, 50257]) Initial Logits: torch.Size([1, 4, 50257]) Initial Past Key Values Shapes: [torch.Size([1, 12, 4, 64]), torch.Size([1, 12, 4, 64]), torch.Size([1, 12, 4, 64]), torch.Size([1, 12, 4, 64]), torch.Size([1, 12, 4, 64]), torch.Size([1, 12, 4, 64]), torch.Size([1, 12, 4, 64]), torch.Size([1, 12, 4, 64]), torch.Size([1, 12, 4, 64]), torch.Size([1, 12, 4, 64]), torch.Size([1, 12, 4, 64]), torch.Size([1, 12, 4, 64])] torch.Size([1, 3, 50257]) Logits: torch.Size([1, 3, 50257]) Past Key Values: 12 pkv: 2 torch.Size([1, 12, 7, 64]) torch.Size([1, 12, 7, 64]) pkv: 2 torch.Size([1, 12, 7, 64]) torch.Size([1, 12, 7, 64]) pkv: 2 torch.Size([1, 12, 7, 64]) torch.Size([1, 12, 7, 64]) pkv: 2 torch.Size([1, 12, 7, 64]) torch.Size([1, 12, 7, 64]) pkv: 2 torch.Size([1, 12, 7, 64]) torch.Size([1, 12, 7, 64]) pkv: 2 torch.Size([1, 12, 7, 64]) torch.Size([1, 12, 7, 64]) pkv: 2 torch.Size([1, 12, 7, 64]) torch.Size([1, 12, 7, 64]) pkv: 2 torch.Size([1, 12, 7, 64]) torch.Size([1, 12, 7, 64]) pkv: 2 torch.Size([1, 12, 7, 64]) torch.Size([1, 12, 7, 64]) pkv: 2 torch.Size([1, 12, 7, 64]) torch.Size([1, 12, 7, 64]) pkv: 2 torch.Size([1, 12, 7, 64]) torch.Size([1, 12, 7, 64]) pkv: 2 torch.Size([1, 12, 7, 64]) torch.Size([1, 12, 7, 64]) torch.Size([1, 7]) torch.Size([1, 3]) torch.Size([1, 12, 4, 64]) Output [198, 40, 1842, 284, 2342] torch.Size([1, 3, 50257]) torch.Size([1, 12]) torch.Size([24, 1, 12, 4, 64]) Second pass: output_id: tensor([[2342]]) new_input_ids: torch.Size([1, 3]) extended_attention_mask: torch.Size([1, 7]) past_key_values: torch.Size([24, 1, 12, 4, 64]) past_key_values_2: torch.Size([24, 1, 12, 7, 64]) torch.Size([1, 5]) /home/ydr/projects/rust-connect-py-ai-to-ic/python/GPT2_kv.py:93: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if torch.all(past_key_values == 0): /home/ydr/anaconda3/lib/python3.11/site-packages/transformers/modeling_attn_mask_utils.py:116: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if (input_shape[-1] > 1 or self.sliding_window is not None) and self.is_causal: /home/ydr/anaconda3/lib/python3.11/site-packages/transformers/modeling_attn_mask_utils.py:164: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if past_key_values_length > 0: [[6332 30 198]] (1, 3) (1, 7) (24, 1, 12, 4, 64) Output ID: [[198]] Past Key Values: (24, 1, 12, 7, 64) [[2061 318 534 4004 6332 30 198]] (1, 7) (1, 7) (12, 2, 1, 12, 1, 64) Traceback (most recent call last): File "/home/ydr/projects/rust-connect-py-ai-to-ic/python/GPT2_kv.py", line 261, in <module> outputs = ort_session.run(None, onnx_inputs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ydr/anaconda3/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 266, in run return self._sess.run(output_names, input_feed, run_options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Invalid rank for input: past_key_values_input Got: 6 Expected: 5 Please fix either the inputs/outputs or the model.
Oh, I asked the chatGPT, and it gave me some advice, and it suggests that the possible cause of this error is the Input Shape of past_key_values,
1 Like
When I provided the onnx file you no longer need to run anything in python. The python files merely download, tweak, and save the GPT2 model to onnx formatting. The onnx file is then uploaded to a deployed canister which runs the tract framework (rust based) and provides a working implementation of GPT2 on chain.
2 Likes
I see… I followed the instructions to step 4 which is to run the GPT_kv.py script here, do you mean I just skip this step and upload the onnx file?
1 Like
Correct. The GPT_kv.py script is run order to generate the onnx file.
2 Likes
Oh~ I really appreciate your answer! Your project is really very groundbreaking!
2 Likes