I’ve been able to get jolt physics sample to run on icp.
It gives the following error though after few steps:
The replica returned a replica error: reject code CanisterError, reject message IC0502: Canister bkyz2-fmaaa-aaaaa-qaaaq-cai trapped: heap out of bounds, error code Some(“IC0502”)
Any thoughts on what might be causing this?
Latest Sample is over here:
Hi @ktimam,
I’ve been working on adding the ability to return backtraces with traps like these and I could try testing it out on your canister to see if it helps narrow down where the error is occurring. Could you provide the compiled Wasm with the name custom section included? (That section should be generated at compilation, so just keep it in by not running optimization tools that strip custom sections).
I might have some clue on where this might be stemming from.
In sCollideShapeVsShape, some static functions are being used. Commenting out the code inside the function removes the issue, while decreasing code inside the function causes the simulation to run for a couple of extra lines of code before breaking again. Seems the use of the static functions is eating up the heap which seems to be more limited in size than usual. Appreciate if someone can confirm this finding and whether there’s a solution or should all those static functions be replaced by normal functions?
Overflowing the stack could trigger a heap out of bounds error (Wasmtime can’t tell that the oob access was actually supposed to be a stack access). I wouldn’t expect making the static functions normal ones would make a difference though.
If you could send a Wasm module with the name section to test maybe we would see a long chain of these calls which could hint at stack overflow.
Thanks for the help. Here is the wasm file I could generate (not sure if it has the name section you need):
Sorry, that wasm has had the “name” section removed. Is your toolchain using something like wasm-strip, ic-wasm shrink, or wasm-opt at the end to make the module smaller? If so, could you provide a copy of the module before any of those tools are run?
One way to verify that the “name” section is present is to install wasm-objdump from WABT and run wasm-objdump -h on your Wasm file. If you see a line like:
Custom start=0x00218d8c end=0x00229def (size=0x00011063) "name"
then it’s still there.
@abk ,
I am trying to also help out with this issue. Thank you for looking into this!
I do not fully understand what you are asking about. Can you try to explain it further?
The wasm was created with icpp, which uses the wasi-sdk to compile C++ to wasm.
No stripping of custom sections takes place, but we also do not add any custom sections.
Where would that “name” section come from and why do you need it?
By default wasi-sdk should be creating the name section when it generates a .wasm file. For example, if I create this C program foo.c:
int main {
return 5;
}
and compile it using the wasi-sdk installed by icpp:
/home/adam/.icpp/wasi-sdk-20.0/bin/clang --sysroot=/home/adam/.icpp/wasi-sdk-20.0/share/wasi-sysroot foo.c -o foo.wasm
then the resulting foo.wasm will have a name section:
❯ wasm-objdump -x -j name foo.wasm
foo.wasm: file format wasm 0x1
Section Details:
Custom:
- name: "name"
- func[0] <__imported_wasi_snapshot_preview1_proc_exit>
- func[1] <__wasm_call_ctors>
- func[2] <_start>
- func[3] <__original_main>
- func[4] <__wasi_proc_exit>
- func[5] <dummy>
- func[6] <__wasm_call_dtors>
- global[0] <__stack_pointer>
I think the -Wl,--strip-all linker flag you’re passing here is telling the linker to skip the name section. If I add that to my example command above:
/home/adam/.icpp/wasi-sdk-20.0/bin/clang --sysroot=/home/adam/.icpp/wasi-sdk-20.0/share/wasi-sysroot -Wl,--strip-all foo.c -o foo.wasm
then the resulting Wasm no longer has the name section:
❯ wasm-objdump -x -j name foo.wasm
foo.wasm: file format wasm 0x1
Section Details:
Section not found: name
So @ktimam, what I think you need to do is modify your installed icpp to take out that flag, or manually run the link command without that flag.
By the way, @icpp I encountered an error when trying to build the hello world example for icpp-pro. Here are the commands I ran:
# create virtualenv and activate it
python -m venv venv
source venv/bin/activate
# install icpp-pro and wasi-sdk
pip install icpp-pro
icpp install-wasi-sdk
# create and build hello-world
icpp init
cd greet/
icpp build-wasm
The versions I have are then:
❯ python --version
Python 3.8.10
❯ icpp --version
icpp-pro version: 3.9.0
wasi-sdk version: wasi-sdk-20.0
And the error I get for the build command is:
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/adam/experiments/icpp/venv/lib/python3.8/site-packages/icpp/decorators.py:63 in decorated │
│ │
│ 60 │ │ │ │ install_wasi_sdk() │
│ 61 │ │ │ │ typer.echo("Now that the wasi-sdk is installed, we can build things.") │
│ 62 │ │ │ │
│ ❱ 63 │ │ │ return f(*args, **kwargs) │
│ 64 │ │ │
│ 65 │ │ return decorated │
│ 66 │
│ │
│ ╭────────────────────────── locals ──────────────────────────╮ │
│ │ args = () │ │
│ │ f = <function build_wasm at 0x7f4a55531790> │ │
│ │ kwargs = {'to_compile': 'all', 'generate_bindings': 'yes'} │ │
│ ╰────────────────────────────────────────────────────────────╯ │
│ │
│ /home/adam/experiments/icpp/venv/lib/python3.8/site-packages/icpp/commands_build_wasm.py:53 in │
│ build_wasm │
│ │
│ 50 │ │
│ 51 │ Reads icpp.toml in the current folder; Compiles & builds a wasm file. │
│ 52 │ """ │
│ ❱ 53 │ from icpp import icpp_toml # pylint: disable = import-outside-toplevel │
│ 54 │ │
│ 55 │ build_path = icpp_toml.icpp_toml_path.parent / "build" │
│ 56 │ typer.echo(f"Build folder: {build_path.resolve()}") │
│ │
│ ╭───────── locals ──────────╮ │
│ │ generate_bindings = 'yes' │ │
│ │ to_compile = 'all' │ │
│ ╰───────────────────────────╯ │
│ │
│ /home/adam/experiments/icpp/venv/lib/python3.8/site-packages/icpp/icpp_toml.py:32 in <module> │
│ │
│ 29 │ sys.exit(1) │
│ 30 │
│ 31 │
│ ❱ 32 def validate(d_in: dict[Any, Any]) -> None: │
│ 33 │ """Validates if required fields are present""" │
│ 34 │ if "build-wasm" not in d_in.keys(): │
│ 35 │ │ typer.echo( │
│ │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │ Any = typing.Any │ │
│ │ glob = <module 'glob' from '/usr/lib/python3.8/glob.py'> │ │
│ │ icpp_toml_path = PosixPath('icpp.toml') │ │
│ │ Path = <class 'pathlib.Path'> │ │
│ │ sys = <module 'sys' (built-in)> │ │
│ │ tomllib = <module 'tomli' from │ │
│ │ '/home/adam/experiments/icpp/venv/lib/python3.8/site-packages/tomli/__init… │ │
│ │ typer = <module 'typer' from │ │
│ │ '/home/adam/experiments/icpp/venv/lib/python3.8/site-packages/typer/__init… │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: 'type' object is not subscriptableThanks for reporting that. I will investigate and fix.
@ktimam ,
I will also push out a new release later this week or in the weekend to allow you to build without that flag, so we can provide @abk with a wasm that includes the name.
I will post a message in this thread once it is available.
I encountered an error when trying to build the hello world example for
icpp-pro.
@abk,
I tested it, and the error does not happen when you use python 3.9 or higher.
I will investigate if I can back-port to python 3.8, or if I will make python 3.9+ a requirement.
Thanks again for reporting this.
Sounds good. Yeah you probably at least want to give an error that the version isn’t compatible.
@abk,
icpp-pro 3.10.0 has been released:
I also updated the ci/cd pipeline to test on python 3.8, so I will not break it again.
Thanks again for reporting the issue!
I have pushed out a release candidate (icpp-pro 3.11.0rc1) which provides fine-grained control over the compile & link flags.
I created an example canister, canister_flags, that uses this icpp.toml and it is not stripping out the information from the wasm file:
[build-wasm]
canister = "my_canister"
did_path = "src/my_canister.did"
cpp_paths = ["src/*.cpp"]
cpp_include_dirs = ["src/vendors/*"]
cpp_compile_flags = ["-D JSON_HAS_FILESYSTEM=0"]
cpp_link_flags = []
c_paths = []
c_include_dirs = []
c_compile_flags = []
# if non-empty, these defaults will overwrite internal settings
# Implemented to support investigation of this issue:
# https://forum.dfinity.org/t/heap-out-of-bounds-error-code-some-ic0502-on-c-code-run/25289/6?u=icpp
cpp_compile_flags_defaults = [
# "-O3",
# "-flto",
"-fno-exceptions", # required for IC
# "-fvisibility=hidden",
"-D NDEBUG",
"-D ICPP_VERBOSE=0",
]
cpp_link_flags_defaults = [
"-nostartfiles",
"-Wl,--no-entry",
# "-Wl,--lto-O3",
# "-Wl,--strip-all",
# "-Wl,--strip-debug",
"-Wl,--stack-first",
"-Wl,--export-dynamic", # required for IC
]
c_compile_flags_defaults = [
# "-O3",
# "-flto",
"-fno-exceptions", # required for IC
# "-fvisibility=hidden",
"-D NDEBUG",
"-D ICPP_VERBOSE
NOTES:
wasm-objdump -h:$ wasm-objdump -h build/my_canister.wasm
my_canister.wasm: file format wasm 0x1
Sections:
Type start=0x0000000b end=0x000001a9 (size=0x0000019e) count: 61
Import start=0x000001ac end=0x00000252 (size=0x000000a6) count: 8
Function start=0x00000256 end=0x0000448d (size=0x00004237) count: 16948
Table start=0x0000448f end=0x00004496 (size=0x00000007) count: 1
Memory start=0x00004498 end=0x0000449b (size=0x00000003) count: 1
Global start=0x0000449e end=0x00004b61 (size=0x000006c3) count: 247
Export start=0x00004b65 end=0x000093b7 (size=0x00004852) count: 505
Elem start=0x000093ba end=0x00009bce (size=0x00000814) count: 1
Code start=0x00009bd3 end=0x00218918 (size=0x0020ed45) count: 16948
Data start=0x0021891c end=0x002248c1 (size=0x0000bfa5) count: 2
Custom start=0x002248c4 end=0x0022521d (size=0x00000959) ".debug_info"
Custom start=0x00225220 end=0x00225b43 (size=0x00000923) ".debug_loc"
Custom start=0x00225b45 end=0x00225b73 (size=0x0000002e) ".debug_ranges"
Custom start=0x00225b76 end=0x00225ff7 (size=0x00000481) ".debug_abbrev"
Custom start=0x00225ffa end=0x002268c1 (size=0x000008c7) ".debug_line"
Custom start=0x002268c4 end=0x00226c44 (size=0x00000380) ".debug_str"
Custom start=0x00226c49 end=0x0113db75 (size=0x00f16f2c) "name"
Custom start=0x0113db77 end=0x0113dbac (size=0x00000035) "producers"
Custom start=0x0113dbae end=0x0113dbe7 (size=0x00000039) "target_features"
WASM created (its bigger than the limit even after zipping though, so couldn’t deploy):
Here’s the backtrace I got when executing the “hello” query:
IC0502: Canister rwlgt-iiaaa-aaaaa-aaaaa-cai trapped: heap out of bounds.
Backtrace:
0: JPH::PhysicsSystem::ProcessBodyPair(JPH::ContactConstraintManager::ContactAllocator&, JPH::BodyPair const&)
1: JPH::PhysicsSystem::JobFindCollisions(JPH::PhysicsUpdateContext::Step*, int)
2: JPH::PhysicsSystem::Update(float, int, JPH::TempAllocator*, JPH::JobSystem*)::$_2::operator()() const
3: decltype(std::declval<JPH::PhysicsSystem::Update(float, int, JPH::TempAllocator*, JPH::JobSystem*)::$_2&>()()) std::__2::__invoke[abi:v160000]<JPH::PhysicsSystem::Update(float, int, JPH::TempAllocator*, JPH::JobSystem*)::$_2&>(JPH::PhysicsSystem::Update(float, int, JPH::TempAllocator*, JPH::JobSystem*)::$_2&)
4: void std::__2::__invoke_void_return_wrapper<void, true>::__call<JPH::PhysicsSystem::Update(float, int, JPH::TempAllocator*, JPH::JobSystem*)::$_2&>(JPH::PhysicsSystem::Update(float, int, JPH::TempAllocator*, JPH::JobSystem*)::$_2&)
5: std::__2::__function::__default_alloc_func<JPH::PhysicsSystem::Update(float, int, JPH::TempAllocator*, JPH::JobSystem*)::$_2, void ()>::operator()[abi:v160000]()
6: void std::__2::__function::__policy_invoker<void ()>::__call_impl<std::__2::__function::__default_alloc_func<JPH::PhysicsSystem::Update(float, int, JPH::TempAllocator*, JPH::JobSystem*)::$_2, void ()>>(std::__2::__function::__policy_storage const*)
7: std::__2::__function::__policy_func<void ()>::operator()[abi:v160000]() const
8: std::__2::function<void ()>::operator()() const
9: JPH::JobSystem::Job::Execute()
10: JPH::JobSystemSingleThreaded::QueueJob(JPH::JobSystem::Job*)
11: JPH::JobSystemSingleThreaded::QueueJobs(JPH::JobSystem::Job**, unsigned int)
12: JPH::JobSystem::JobHandle::sRemoveDependencies(JPH::JobSystem::JobHandle const*, unsigned int, int)
13: void JPH::JobSystem::JobHandle::sRemoveDependencies<32u>(JPH::StaticArray<JPH::JobSystem::JobHandle, 32u>&, int)
14: JPH::PhysicsSystem::Update(float, int, JPH::TempAllocator*, JPH::JobSystem*)
15: canister_query hello
16: canister_query hello.command_export
So it looks like the issue is in ProcessBodyPair. I can look at it a bit more later.
Thanks for the trace. Delving deeper you’ll find that it breaks in CollisionDispatch::sCollideShapeVsShape, which is a static function, which won’t appear in the trace. When analyzing CollisionDispatch::sCollideShapeVsShape, it is accessing sCollideShape, which is a function pointer table (another use of static functions). All of this use of static functions is my deduction for why it gives out of heap error.
Is there a way to run dfx without the heap limitation to check if the program would run fine?
I can see if it’s possible to modify dfx to have a larger stack.
But how do you know that the oob access is happening in sCollideShapeVsShape? Looking at the generated Wasm that function isn’t actually being inlined so I think it should show up in the backtrace if that’s where the error occured.
Here’s where the call to sCollideShapeVsShape happens in ProcessBodyPair:
❯ wasm-objdump -d joltsample.wasm
joltsample.wasm: file format wasm 0x1
Code Disassembly:
...
59425c func[11073] <JPH::PhysicsSystem::ProcessBodyPair(JPH::ContactConstraintManager::ContactAllocator&, JPH::BodyPair const&)>:
59425e: 67 7f | local[0..102] type=i32
594260: 01 7d | local[103] type=f32
...
595fea: 20 dd 04 | local.get 605
595fed: 20 df 04 | local.get 607
595ff0: 10 aa 90 80 80 00 | call 2090 <JPH::CollisionDispatch::sCollideShapeVsShape(JPH::Shape const*, JPH::Shape const*, JPH::Vec3, JPH::Vec3, JPH::Mat44 const&, JPH::Mat44 const&, JPH::SubSha
peIDCreator const&, JPH::SubShapeIDCreator const&, JPH::CollideShapeSettings const&, JPH::CollisionCollector<JPH::CollideShapeResult, JPH::CollisionCollectorTraitsCollideShape>&, JPH::ShapeFilter const&)>
595ff6: 41 88 02 | i32.const 264
595ff9: 21 e0 04 | local.set 608
595ffc: 20 05 | local.get 5