# Completed: ICDevs.org Bounty #8 - HttpRequest Parser

**URL:** <https://forum.dfinity.org/t/completed-icdevs-org-bounty-8-httprequest-parser/10219>\
**Category:** Bounties & RFPs\
**Created:** [January 11, 2022, 5:29pm UTC](https://forum.dfinity.org/t/completed-icdevs-org-bounty-8-httprequest-parser/10219 "2022-01-11T17:29:08Z")\
**Posts on this page:** 15\
**Page:** 2

<div class="post-metadata">

**Author:** ![skilesare](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/skilesare/32/5609_2.png) [@skilesare](https://forum.dfinity.org/u/skilesare)\
**Post date:** [April 1, 2022, 1:37pm UTC](https://forum.dfinity.org/t/completed-icdevs-org-bounty-8-httprequest-parser/10219/21 "2022-04-01T13:37:59Z")

</div>

I’d suggest switching to triemap as hashmap has serious memory issues.

---

<div class="post-metadata">

**Author:** ![tomijaga](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/tomijaga/32/4389_2.png) [@tomijaga](https://forum.dfinity.org/u/tomijaga)\
**Post date:** [April 2, 2022, 6:28am UTC](https://forum.dfinity.org/t/completed-icdevs-org-bounty-8-httprequest-parser/10219/22 "2022-04-02T06:28:30Z")

</div>

> Do you happen to have some examples that demonstrate how to use it?

Yes, I have an [example canister](https://github.com/tomijaga/http-parser.mo/tree/main/example) in the repo with a simple HTML form for uploading a file and some fields. You can clone the repo and deploy it locally to access the form at [http://localhost:8000](http://localhost:8000). I have a function called [debugRequestParser](https://github.com/tomijaga/http-parser.mo/blob/main/example/src/main.mo#L15) that prints the parsed request to the console once the form is submitted. This function should show how to use and access some fields in the object.

> How does one actually call it when it is deployed to either the local network or the IC?

The parser is a module that can be imported into your canister by adding the `.vessel.dhall` and `package-set.dhall` files specified in the example canister and this line, `import HttpParser "mo:HttpParser";`, to your code. It would be called in the `http_request` function on the incoming request to the canister. Here’s a snippet of what it would look like:

```motoko
    public query func http_request(rawReq: HttpParser.HttpRequest) : async HttpParser.HttpResponse {

        let req = HttpParser.parse(rawReq);

        let {host; port; path; queryObj; anchor; original = url} = req.url;

               ...

```

---

<div class="post-metadata">

**Author:** ![tomijaga](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/tomijaga/32/4389_2.png) [@tomijaga](https://forum.dfinity.org/u/tomijaga)\
**Post date:** [April 2, 2022, 7:55am UTC](https://forum.dfinity.org/t/completed-icdevs-org-bounty-8-httprequest-parser/10219/23 "2022-04-02T07:55:59Z")

</div>

Thanks, I will switch to that. Do you have any other suggestions on how to be more efficient? Parsing files over 30kb is really slow. It takes about 40s. I think it’s because I’m concatenating the characters in every line when I only need to check the first character for a match and move to the next line if it fails. I will try this out and get back to you about the performance improvements if there are any.

---

<div class="post-metadata">

**Author:** ![tomijaga](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/tomijaga/32/4389_2.png) [@tomijaga](https://forum.dfinity.org/u/tomijaga)\
**Post date:** [April 2, 2022, 10:52am UTC](https://forum.dfinity.org/t/completed-icdevs-org-bounty-8-httprequest-parser/10219/24 "2022-04-02T10:52:52Z")

</div>

I just implemented this, and there were some improvements. The module now parses data at 100kb/s, which is still relatively slow as it takes about 10 hours to parse a 3GB file. I will check other similar parsing libs to see how I can increase the performance further.

---

<div class="post-metadata">

**Author:** ![skilesare](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/skilesare/32/5609_2.png) [@skilesare](https://forum.dfinity.org/u/skilesare)\
**Post date:** [April 2, 2022, 2:19pm UTC](https://forum.dfinity.org/t/completed-icdevs-org-bounty-8-httprequest-parser/10219/25 "2022-04-02T14:19:35Z")

</div>

Wow. We are going to need a strategy for that. We really need a highly performant regex function. I’ll try to take a look, but perhaps @paulyoung has an idea?

---

<div class="post-metadata">

**Author:** ![skilesare](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/skilesare/32/5609_2.png) [@skilesare](https://forum.dfinity.org/u/skilesare)\
**Post date:** [April 3, 2022, 12:40am UTC](https://forum.dfinity.org/t/completed-icdevs-org-bounty-8-httprequest-parser/10219/26 "2022-04-03T00:40:16Z")

</div>

One thing to keep in mind is that requests on the IC are limited to 2MB, so you are unlikely to run into a scenario where you need to parse more than that. I think this applies to http\_request as well.

If you want a file bigger than that, you have to chunk it.

---

<div class="post-metadata">

**Author:** ![paulyoung](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/paulyoung/32/5251_2.png) [@paulyoung](https://forum.dfinity.org/u/paulyoung)\
**Post date:** [April 5, 2022, 2:21am UTC](https://forum.dfinity.org/t/completed-icdevs-org-bounty-8-httprequest-parser/10219/27 "2022-04-05T02:21:53Z")

</div>

> [@skilesare](#):
>
> Wow. We are going to need a strategy for that. We really need a highly performant regex function. I’ll try to take a look, but perhaps @paulyoung has an idea?

As I said earlier in this thread; I suggest using parser combinators, or at least a parser that consumes the input as it goes.

I haven’t used these but they might be a good place to start.

> **[GitHub - crusso/mo-parsec: A parsec parser combinator library for Motoko](https://github.com/crusso/mo-parsec)**
>
> A parsec parser combinator library for Motoko. Contribute to crusso/mo-parsec development by creating an account on GitHub.

> **[GitHub - aviate-labs/parser-combinators.mo: Parser Combinator Library for Motoko](https://github.com/aviate-labs/parser-combinators.mo)**
>
> Parser Combinator Library for Motoko. Contribute to aviate-labs/parser-combinators.mo development by creating an account on GitHub.

---

<div class="post-metadata">

**Author:** ![skilesare](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/skilesare/32/5609_2.png) [@skilesare](https://forum.dfinity.org/u/skilesare)\
**Post date:** [April 15, 2022, 1:55am UTC](https://forum.dfinity.org/t/completed-icdevs-org-bounty-8-httprequest-parser/10219/28 "2022-04-15T01:55:48Z")

</div>

Any new updates? Looking forward to paying out the bounty!

---

<div class="post-metadata">

**Author:** ![tomijaga](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/tomijaga/32/4389_2.png) [@tomijaga](https://forum.dfinity.org/u/tomijaga)\
**Post date:** [April 16, 2022, 9:18pm UTC](https://forum.dfinity.org/t/completed-icdevs-org-bounty-8-httprequest-parser/10219/29 "2022-04-16T21:18:30Z")

</div>

Yes, I’ve made a few updates. I’ve added support for percent-encoded search queries, written [unit tests](https://github.com/tomijaga/http-parser.mo/blob/main/tests/Test.mo) for each class and added [documentation](https://github.com/tomijaga/http-parser.mo/blob/main/docs.md) for the ParsedHttpRequest data type.

However, I haven’t been able to get the module to parse files faster. I’ve tried looking into parser combinators (thanks @paulyoung for this btw), but I haven’t been able to wrap my head around them. It will take some time to understand how they work and use them in the module.

---

<div class="post-metadata">

**Author:** ![skilesare](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/skilesare/32/5609_2.png) [@skilesare](https://forum.dfinity.org/u/skilesare)\
**Post date:** [April 16, 2022, 9:41pm UTC](https://forum.dfinity.org/t/completed-icdevs-org-bounty-8-httprequest-parser/10219/30 "2022-04-16T21:41:56Z")

</div>

Please push your changes as they are already super useful. There are better ways to get files into the IC than http\_request anyway. We can always explore and add a version later.

---

<div class="post-metadata">

**Author:** ![paulyoung](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/paulyoung/32/5251_2.png) [@paulyoung](https://forum.dfinity.org/u/paulyoung)\
**Post date:** [April 16, 2022, 11:12pm UTC](https://forum.dfinity.org/t/completed-icdevs-org-bounty-8-httprequest-parser/10219/31 "2022-04-16T23:12:11Z")

</div>

This is a beginner tutorial in Rust but I think it describes some fundamental concepts well. I would focus on the “Defining the parser” and “Combinators” sections.

[https://bodil.lol/parser-combinators/](https://bodil.lol/parser-combinators/)

Something like this might be closer in syntax: [Building Parser Combinators (Part 1) - Swift Talk - objc.io](https://talk.objc.io/episodes/S01E15-building-parser-combinators-part-1)

---

<div class="post-metadata">

**Author:** ![tomijaga](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/tomijaga/32/4389_2.png) [@tomijaga](https://forum.dfinity.org/u/tomijaga)\
**Post date:** [May 2, 2022, 3:00am UTC](https://forum.dfinity.org/t/completed-icdevs-org-bounty-8-httprequest-parser/10219/32 "2022-05-02T03:00:28Z")

</div>

Hey @skilesare , I have updated the repo with my changes and completed the development for this version. I believe the package is ready to be reviewed and would be awaiting your feedback on any other improvements that could be made.

---

<div class="post-metadata">

**Author:** ![skilesare](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/skilesare/32/5609_2.png) [@skilesare](https://forum.dfinity.org/u/skilesare)\
**Post date:** [May 2, 2022, 3:25am UTC](https://forum.dfinity.org/t/completed-icdevs-org-bounty-8-httprequest-parser/10219/33 "2022-05-02T03:25:06Z")

</div>

Fantastic. This bounty is officially in preview mode.

We are using a fork in the Origyn NFT already! I’ll review and update to this.

Community: Please review @tomijaga work. This is a very cool library that will save you a ton of time when interpreting http\_request queries!

---

<div class="post-metadata">

**Author:** ![skilesare](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/skilesare/32/5609_2.png) [@skilesare](https://forum.dfinity.org/u/skilesare)\
**Post date:** [May 16, 2022, 8:07pm UTC](https://forum.dfinity.org/t/completed-icdevs-org-bounty-8-httprequest-parser/10219/34 "2022-05-16T20:07:54Z")

</div>

This bounty is now closed and awarded. Congrats @tomijaga!

---

<div class="post-metadata">

**Author:** ![tomijaga](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.dfinity.org/tomijaga/32/4389_2.png) [@tomijaga](https://forum.dfinity.org/u/tomijaga)\
**Post date:** [May 22, 2022, 7:43am UTC](https://forum.dfinity.org/t/completed-icdevs-org-bounty-8-httprequest-parser/10219/35 "2022-05-22T07:43:07Z")

</div>

Thank you, @skilesare and everyone in this forum for your contributions and support.  
I will continue to maintain the repo and work on a more performant version in my free time.  
If you are interested in using this lib in your project, you can find the repo here

> [@Completed: ICDevs.org Bounty #8 - HttpRequest Parser](https://forum.dfinity.org/t/icdevs-org-bounty-8-httprequest-parser/10219/15):
>
> My bad, I’ve made it public now. [GitHub - tomijaga/http-parser.mo: HTTP Request Parser for Motoko](https://github.com/tomijaga/http-parser.mo)

[Previous page](https://forum.dfinity.org/t/completed-icdevs-org-bounty-8-httprequest-parser/10219.md?page=1)