ktimam
December 5, 2023, 5:50pm
1
I’ve been able to get jolt physics sample to run on icp.
It gives the following error though after few steps:
The replica returned a replica error: reject code CanisterError, reject message IC0502: Canister bkyz2-fmaaa-aaaaa-qaaaq-cai trapped: heap out of bounds, error code Some(“IC0502”)
Any thoughts on what might be causing this?
Latest Sample is over here:
2 Likes
abk
December 6, 2023, 11:57am
2
Hi @ktimam ,
I’ve been working on adding the ability to return backtraces with traps like these and I could try testing it out on your canister to see if it helps narrow down where the error is occurring. Could you provide the compiled Wasm with the name
custom section included? (That section should be generated at compilation, so just keep it in by not running optimization tools that strip custom sections).
1 Like
ktimam
December 6, 2023, 2:01pm
3
I might have some clue on where this might be stemming from.
In sCollideShapeVsShape, some static functions are being used. Commenting out the code inside the function removes the issue, while decreasing code inside the function causes the simulation to run for a couple of extra lines of code before breaking again. Seems the use of the static functions is eating up the heap which seems to be more limited in size than usual. Appreciate if someone can confirm this finding and whether there’s a solution or should all those static functions be replaced by normal functions?
static inline void sCollideShapeVsShape(const Shape *inShape1, const Shape *inShape2, Vec3Arg inScale1, Vec3Arg inScale2, Mat44Arg inCenterOfMassTransform1, Mat44Arg inCenterOfMassTransform2, const SubShapeIDCreator &inSubShapeIDCreator1, const SubShapeIDCreator &inSubShapeIDCreator2, const CollideShapeSettings &inCollideShapeSettings, CollideShapeCollector &ioCollector, const ShapeFilter &inShapeFilter = { })
1 Like
abk
December 6, 2023, 3:20pm
4
Overflowing the stack could trigger a heap out of bounds error (Wasmtime can’t tell that the oob access was actually supposed to be a stack access). I wouldn’t expect making the static
functions normal ones would make a difference though.
If you could send a Wasm module with the name section to test maybe we would see a long chain of these calls which could hint at stack overflow.
1 Like
ktimam
December 7, 2023, 7:34pm
5
Thanks for the help. Here is the wasm file I could generate (not sure if it has the name section you need):
Shared with Dropbox
1 Like
abk
December 11, 2023, 10:10am
6
Sorry, that wasm has had the “name” section removed. Is your toolchain using something like wasm-strip
, ic-wasm shrink
, or wasm-opt
at the end to make the module smaller? If so, could you provide a copy of the module before any of those tools are run?
One way to verify that the “name” section is present is to install wasm-objdump
from WABT and run wasm-objdump -h
on your Wasm file. If you see a line like:
Custom start=0x00218d8c end=0x00229def (size=0x00011063) "name"
then it’s still there.
icpp
December 12, 2023, 2:44am
7
@abk ,
I am trying to also help out with this issue. Thank you for looking into this!
I do not fully understand what you are asking about. Can you try to explain it further?
The wasm was created with icpp, which uses the wasi-sdk to compile C++ to wasm.
No stripping of custom sections takes place, but we also do not add any custom sections.
Where would that “name” section come from and why do you need it?
1 Like
abk
December 12, 2023, 9:16am
8
By default wasi-sdk
should be creating the name section when it generates a .wasm
file. For example, if I create this C program foo.c
:
int main {
return 5;
}
and compile it using the wasi-sdk
installed by icpp
:
/home/adam/.icpp/wasi-sdk-20.0/bin/clang --sysroot=/home/adam/.icpp/wasi-sdk-20.0/share/wasi-sysroot foo.c -o foo.wasm
then the resulting foo.wasm
will have a name section:
❯ wasm-objdump -x -j name foo.wasm
foo.wasm: file format wasm 0x1
Section Details:
Custom:
- name: "name"
- func[0] <__imported_wasi_snapshot_preview1_proc_exit>
- func[1] <__wasm_call_ctors>
- func[2] <_start>
- func[3] <__original_main>
- func[4] <__wasi_proc_exit>
- func[5] <dummy>
- func[6] <__wasm_call_dtors>
- global[0] <__stack_pointer>
I think the -Wl,--strip-all
linker flag you’re passing here is telling the linker to skip the name section. If I add that to my example command above:
/home/adam/.icpp/wasi-sdk-20.0/bin/clang --sysroot=/home/adam/.icpp/wasi-sdk-20.0/share/wasi-sysroot -Wl,--strip-all foo.c -o foo.wasm
then the resulting Wasm no longer has the name section:
❯ wasm-objdump -x -j name foo.wasm
foo.wasm: file format wasm 0x1
Section Details:
Section not found: name
So @ktimam , what I think you need to do is modify your installed icpp
to take out that flag, or manually run the link command without that flag.
2 Likes
abk
December 12, 2023, 9:24am
9
By the way, @icpp I encountered an error when trying to build the hello world example for icpp-pro
. Here are the commands I ran:
# create virtualenv and activate it
python -m venv venv
source venv/bin/activate
# install icpp-pro and wasi-sdk
pip install icpp-pro
icpp install-wasi-sdk
# create and build hello-world
icpp init
cd greet/
icpp build-wasm
The versions I have are then:
❯ python --version
Python 3.8.10
❯ icpp --version
icpp-pro version: 3.9.0
wasi-sdk version: wasi-sdk-20.0
And the error I get for the build command is:
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/adam/experiments/icpp/venv/lib/python3.8/site-packages/icpp/decorators.py:63 in decorated │
│ │
│ 60 │ │ │ │ install_wasi_sdk() │
│ 61 │ │ │ │ typer.echo("Now that the wasi-sdk is installed, we can build things.") │
│ 62 │ │ │ │
│ ❱ 63 │ │ │ return f(*args, **kwargs) │
│ 64 │ │ │
│ 65 │ │ return decorated │
│ 66 │
│ │
│ ╭────────────────────────── locals ──────────────────────────╮ │
│ │ args = () │ │
│ │ f = <function build_wasm at 0x7f4a55531790> │ │
│ │ kwargs = {'to_compile': 'all', 'generate_bindings': 'yes'} │ │
│ ╰────────────────────────────────────────────────────────────╯ │
│ │
│ /home/adam/experiments/icpp/venv/lib/python3.8/site-packages/icpp/commands_build_wasm.py:53 in │
│ build_wasm │
│ │
│ 50 │ │
│ 51 │ Reads icpp.toml in the current folder; Compiles & builds a wasm file. │
│ 52 │ """ │
│ ❱ 53 │ from icpp import icpp_toml # pylint: disable = import-outside-toplevel │
│ 54 │ │
│ 55 │ build_path = icpp_toml.icpp_toml_path.parent / "build" │
│ 56 │ typer.echo(f"Build folder: {build_path.resolve()}") │
│ │
│ ╭───────── locals ──────────╮ │
│ │ generate_bindings = 'yes' │ │
│ │ to_compile = 'all' │ │
│ ╰───────────────────────────╯ │
│ │
│ /home/adam/experiments/icpp/venv/lib/python3.8/site-packages/icpp/icpp_toml.py:32 in <module> │
│ │
│ 29 │ sys.exit(1) │
│ 30 │
│ 31 │
│ ❱ 32 def validate(d_in: dict[Any, Any]) -> None: │
│ 33 │ """Validates if required fields are present""" │
│ 34 │ if "build-wasm" not in d_in.keys(): │
│ 35 │ │ typer.echo( │
│ │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │ Any = typing.Any │ │
│ │ glob = <module 'glob' from '/usr/lib/python3.8/glob.py'> │ │
│ │ icpp_toml_path = PosixPath('icpp.toml') │ │
│ │ Path = <class 'pathlib.Path'> │ │
│ │ sys = <module 'sys' (built-in)> │ │
│ │ tomllib = <module 'tomli' from │ │
│ │ '/home/adam/experiments/icpp/venv/lib/python3.8/site-packages/tomli/__init… │ │
│ │ typer = <module 'typer' from │ │
│ │ '/home/adam/experiments/icpp/venv/lib/python3.8/site-packages/typer/__init… │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: 'type' object is not subscriptable
1 Like
icpp
December 13, 2023, 1:09am
10
Thanks for reporting that. I will investigate and fix.
1 Like
icpp
December 13, 2023, 2:46am
11
@ktimam ,
I will also push out a new release later this week or in the weekend to allow you to build without that flag, so we can provide @abk with a wasm that includes the name.
I will post a message in this thread once it is available.
2 Likes
icpp
December 13, 2023, 12:28pm
12
I encountered an error when trying to build the hello world example for icpp-pro
.
@abk ,
I tested it, and the error does not happen when you use python 3.9 or higher.
I will investigate if I can back-port to python 3.8, or if I will make python 3.9+ a requirement.
Thanks again for reporting this.
1 Like
abk
December 13, 2023, 1:40pm
13
Sounds good. Yeah you probably at least want to give an error that the version isn’t compatible.
icpp
December 16, 2023, 4:02pm
14
@abk ,
icpp-pro 3.10.0 has been released:
it works again for python 3.8
you cannot install with python 3.7 or earlier
I also updated the ci/cd pipeline to test on python 3.8, so I will not break it again.
Thanks again for reporting the issue!
1 Like
icpp
December 18, 2023, 9:12pm
15
@ktimam (CC. @abk )
I have pushed out a release candidate (icpp-pro 3.11.0rc1
) which provides fine-grained control over the compile & link flags.
I created an example canister, canister_flags , that uses this icpp.toml
and it is not stripping out the information from the wasm file:
[build-wasm]
canister = "my_canister"
did_path = "src/my_canister.did"
cpp_paths = ["src/*.cpp"]
cpp_include_dirs = ["src/vendors/*"]
cpp_compile_flags = ["-D JSON_HAS_FILESYSTEM=0"]
cpp_link_flags = []
c_paths = []
c_include_dirs = []
c_compile_flags = []
# if non-empty, these defaults will overwrite internal settings
# Implemented to support investigation of this issue:
# https://forum.dfinity.org/t/heap-out-of-bounds-error-code-some-ic0502-on-c-code-run/25289/6?u=icpp
cpp_compile_flags_defaults = [
# "-O3",
# "-flto",
"-fno-exceptions", # required for IC
# "-fvisibility=hidden",
"-D NDEBUG",
"-D ICPP_VERBOSE=0",
]
cpp_link_flags_defaults = [
"-nostartfiles",
"-Wl,--no-entry",
# "-Wl,--lto-O3",
# "-Wl,--strip-all",
# "-Wl,--strip-debug",
"-Wl,--stack-first",
"-Wl,--export-dynamic", # required for IC
]
c_compile_flags_defaults = [
# "-O3",
# "-flto",
"-fno-exceptions", # required for IC
# "-fvisibility=hidden",
"-D NDEBUG",
"-D ICPP_VERBOSE
NOTES:
2 Likes
abk
December 19, 2023, 9:28am
16
That looks great. Would you be able to build a version of your canister with that release @ktimam ?
ktimam
December 19, 2023, 11:03am
17
WASM created (its bigger than the limit even after zipping though, so couldn’t deploy):
Shared with Dropbox
3 Likes
abk
December 20, 2023, 3:43pm
18
Here’s the backtrace I got when executing the “hello” query:
IC0502: Canister rwlgt-iiaaa-aaaaa-aaaaa-cai trapped: heap out of bounds.
Backtrace:
0: JPH::PhysicsSystem::ProcessBodyPair(JPH::ContactConstraintManager::ContactAllocator&, JPH::BodyPair const&)
1: JPH::PhysicsSystem::JobFindCollisions(JPH::PhysicsUpdateContext::Step*, int)
2: JPH::PhysicsSystem::Update(float, int, JPH::TempAllocator*, JPH::JobSystem*)::$_2::operator()() const
3: decltype(std::declval<JPH::PhysicsSystem::Update(float, int, JPH::TempAllocator*, JPH::JobSystem*)::$_2&>()()) std::__2::__invoke[abi:v160000]<JPH::PhysicsSystem::Update(float, int, JPH::TempAllocator*, JPH::JobSystem*)::$_2&>(JPH::PhysicsSystem::Update(float, int, JPH::TempAllocator*, JPH::JobSystem*)::$_2&)
4: void std::__2::__invoke_void_return_wrapper<void, true>::__call<JPH::PhysicsSystem::Update(float, int, JPH::TempAllocator*, JPH::JobSystem*)::$_2&>(JPH::PhysicsSystem::Update(float, int, JPH::TempAllocator*, JPH::JobSystem*)::$_2&)
5: std::__2::__function::__default_alloc_func<JPH::PhysicsSystem::Update(float, int, JPH::TempAllocator*, JPH::JobSystem*)::$_2, void ()>::operator()[abi:v160000]()
6: void std::__2::__function::__policy_invoker<void ()>::__call_impl<std::__2::__function::__default_alloc_func<JPH::PhysicsSystem::Update(float, int, JPH::TempAllocator*, JPH::JobSystem*)::$_2, void ()>>(std::__2::__function::__policy_storage const*)
7: std::__2::__function::__policy_func<void ()>::operator()[abi:v160000]() const
8: std::__2::function<void ()>::operator()() const
9: JPH::JobSystem::Job::Execute()
10: JPH::JobSystemSingleThreaded::QueueJob(JPH::JobSystem::Job*)
11: JPH::JobSystemSingleThreaded::QueueJobs(JPH::JobSystem::Job**, unsigned int)
12: JPH::JobSystem::JobHandle::sRemoveDependencies(JPH::JobSystem::JobHandle const*, unsigned int, int)
13: void JPH::JobSystem::JobHandle::sRemoveDependencies<32u>(JPH::StaticArray<JPH::JobSystem::JobHandle, 32u>&, int)
14: JPH::PhysicsSystem::Update(float, int, JPH::TempAllocator*, JPH::JobSystem*)
15: canister_query hello
16: canister_query hello.command_export
So it looks like the issue is in ProcessBodyPair
. I can look at it a bit more later.
ktimam
December 20, 2023, 9:38pm
19
abk:
ProcessBodyPair
Thanks for the trace. Delving deeper you’ll find that it breaks in CollisionDispatch::sCollideShapeVsShape, which is a static function, which won’t appear in the trace. When analyzing CollisionDispatch::sCollideShapeVsShape, it is accessing sCollideShape, which is a function pointer table (another use of static functions). All of this use of static functions is my deduction for why it gives out of heap error.
Is there a way to run dfx without the heap limitation to check if the program would run fine?
abk
December 21, 2023, 9:34am
20
I can see if it’s possible to modify dfx to have a larger stack.
But how do you know that the oob access is happening in sCollideShapeVsShape
? Looking at the generated Wasm that function isn’t actually being inlined so I think it should show up in the backtrace if that’s where the error occured.
Here’s where the call to sCollideShapeVsShape
happens in ProcessBodyPair
:
❯ wasm-objdump -d joltsample.wasm
joltsample.wasm: file format wasm 0x1
Code Disassembly:
...
59425c func[11073] <JPH::PhysicsSystem::ProcessBodyPair(JPH::ContactConstraintManager::ContactAllocator&, JPH::BodyPair const&)>:
59425e: 67 7f | local[0..102] type=i32
594260: 01 7d | local[103] type=f32
...
595fea: 20 dd 04 | local.get 605
595fed: 20 df 04 | local.get 607
595ff0: 10 aa 90 80 80 00 | call 2090 <JPH::CollisionDispatch::sCollideShapeVsShape(JPH::Shape const*, JPH::Shape const*, JPH::Vec3, JPH::Vec3, JPH::Mat44 const&, JPH::Mat44 const&, JPH::SubSha
peIDCreator const&, JPH::SubShapeIDCreator const&, JPH::CollideShapeSettings const&, JPH::CollisionCollector<JPH::CollideShapeResult, JPH::CollisionCollectorTraitsCollideShape>&, JPH::ShapeFilter const&)>
595ff6: 41 88 02 | i32.const 264
595ff9: 21 e0 04 | local.set 608
595ffc: 20 05 | local.get 5