Implementation of xxHash in Motoko as a module

Developers Programs & Applications
1

Hi,

I need to generate hash from large files for one of my projects. Faced with size limitations and exception like “exceeded the instruction limit for single message execution”.

I have implemented the xxhash algorithm. It is currently available for motoko in two versions, 32 and 64 bits.
It allows me, to:

  • Easily check with hashes generated from other systems like xxhsum

  • Process large MB files in several times and thus simplify the management of the 2MB limit for parameters

  • Easily and quickly process data smaller than 2MB

This algorithm is not cryptographic but it has good performance and quality characteristics (more)

The source code is available gitlab

The tests show that the hashes are correct, I’m still trying to find ways to decrease the number of instructions consumed.

Your feedback is welcome

3 Likes
2

Hey @rbolog,

Thanks for sharing this library. I needed it for uniform distribution, and It saved me time that I would have otherwise spent building something from scratch.

I made some updates to the code to improve performance and reduce the number of instructions executed.
Specifically, I replaced the Buffer classes with a global fixed-sized array to prevent constant allocation and updated the lane variable to a single Nat64 type that cumulatively appends input bytes.

You can find the updated version of the code in the xxHashV2 file at the following link:

I also added a function to compare the performance between the two versions.

1 Like
3

Hi @tomijaga

Thanks for sharing your updates, I will integrate it soon.

1 Like