Exposing URLs for querying images

kevinli · October 3, 2021, 7:40am

I’m interested in exposing a url from a canister where someone can send an http request to it and receive back an image. I was wondering how to do something like this:
https://e3izy-jiaaa-aaaah-qacbq-cai.raw.ic0.app/?tokenid=onxj3-rakor-uwiaa-aaaaa-b4aaq-maqca-aabc7-a
If I have a backend/storage canister that stores a bunch of images as Blobs, is it possible to process these requests and return those Blobs as deserialized images?
Or do I need to create a separate frontend that queries the backend for the Blob, deserializes the images there, and then returns the image?

spencer · October 3, 2021, 3:53pm

I’ve been looking into this as well, and have decided to go with the latter approach of returning all images and videos as blobs first retrieved to a local cache on the browser then calling createobjecturl to render the images.

Out of curiosity, what project is that image link from? How did they do that?

nomeata · October 3, 2021, 5:31pm

Is your backend canister implemented in rust or motoko?

But in both cases it is not hard to return raw HTTP, by exposing the http_query method, at least if you are content with <canisterid>.raw.ic0.app URLs and the fact that this is not safe against malicoius nodes.

There just isn’t much documentation yet, and not even the spec is out yet, but you cargo-cult from various example projects.

Serving it in a way that works from <cansiterid>.ic0.app (i.e. certified) is also possible, but significantly harder.

Or you can just use dfx’s default “certified asset canister” to store the files, then HTTP access works out of the box. Again, I believe we are missing a tutorial for how to use that canister in your poject, e.g. upload files to it.

spencer · October 3, 2021, 6:11pm

@nomeata Do you know of any such example projects? I’m using motoko backend.

nomeata · October 3, 2021, 9:08pm

GitHub - nomeata/motoko-certified-http: A motoko canister that can server certified HTTP assets and you can ignore the certification stuff I guess.

kevinli · October 3, 2021, 9:59pm

@nomeata
Interesting, so if I basically just include this part:

public query func http_request(req : HttpRequest) : async HttpResponse {
    // check if / is requested
    if ((req.method, req.url) == ("GET", "/")) {
      // If so, return the main page with with right headers
      return {
        status_code = 200;
        headers = [ ("content-type", "text/plain"), certification_header() ];
        body = main_page()
      }
    } else {
      // Else return an error code. Note that we cannot certify this response
      // so a user going to https://ce7vw-haaaa-aaaai-aanva-cai.ic0.app/foo
      // will not see the error message
      return {
        status_code = 404;
        headers = [ ("content-type", "text/plain") ];
        body = "404 Not found.\n This canister only serves /.\n"
      }
    }
  };

the assets will be queryable simply by doing <canisterid>.raw.ic0.app?

nomeata · October 4, 2021, 7:32am

That will serve the content of main_page() at https://<canisterid>.raw.ic0.app/, as you can see at https://ce7vw-haaaa-aaaai-aanva-cai.ic0.app/.

But you want to serve images, and probably at different URLs. So you can add something like

    else if ((req.method, req.url) == ("GET", "/my_image.png")) {
      return {
        status_code = 200;
        headers = [ ("content-type", "image/png") ];
        body = the_image_blob
      }

And if it’s not a fixed set of images, your URL routing logic becomes more complicated…

Also, this code doesn’t do streaming of HTTP responses, so don’t expect it to work with files larger than 2MB.

Another source for inspiration could be the “old” asset canister, as it used to come with dfx. That’s also written in Motoko, and does more complicated things with assets. It got removed from the sdk repo, but we have history: https://github.com/dfinity/sdk/tree/b334a69408e7fc873e09d2d82c3fe827665fcfa1/src/distributed/assetstorage.

kevinli · October 4, 2021, 7:36am

@nomeata
Ah, okay, so how would one pass in a path variable and parse that?
Let’s say I want something like:

else if ((req.method, req.url) == ("GET", "/images/:imageId")) {
      return {
        status_code = 200;
        headers = [ ("content-type", "image/png") ];
        body = { grab the correct blob from storage with imageId }
      }

How could I achieve this with a dynamic path variable?

nomeata · October 4, 2021, 9:19am

You probably want to analyze req.url with Text.stripStart to see if it starts with /images/ and get the remainder… there is no URL routing library for Motoko yet so you have to implement that yourself. But it’s just “plain programming” at this point

cryptoschindler · October 4, 2021, 11:36am

If we were to use a custom implemented asset canister that exposes the same interface as the one provided by the SDK, can we still use dfx deploy to upload assets into it if we specify type : assets?

I’m not too familiar with rust, can you maybe explain where the ic_asset here is coming from?

github.com

dfinity/sdk/blob/dacd7ecb1dc64fd7113ea84eea406596a6390eb7/src/dfx/src/lib/installers/assets/mod.rs#L23

    
      
              let assets_canister_info = info.as_info::<AssetsCanisterInfo>()?;
              let output_assets_path = assets_canister_info.get_output_assets_path();
          
          
    let canister_id = info.get_canister_id().expect("Could not find canister ID.");
          
          
    let canister = ic_utils::Canister::builder()
                  .with_agent(&agent)
                  .with_canister_id(canister_id)
                  .build()?;
          
          
    ic_asset::sync(&canister, output_assets_path, timeout).await?;
          
          
    Ok(())
          }

nomeata · October 4, 2021, 1:57pm

I would assume so.

Not at first glance. Maybe @roman-kashitsyn can help.

nomeata · October 4, 2021, 5:06pm

I just came across this code, which is probably a better example for serving HTTP requests from Motoko:

github.com

ORIGYN-SA/motoko_http_handler/blob/master/src/motoko_http_handler/main.mo#L56

    
      
                  token: StreamingCallbackToken;
              };
          };
          
          
public type HeaderField = (Text, Text);
          
          
private func removeQuery(str: Text): Text {
              return Option.unwrap(Text.split(str, #char '?').next());
          };
          
          
public query func http_request(req: HttpRequest): async (HttpResponse) {
              let path = removeQuery(req.url);
              
              if(path == "/") {
                  return {
                      body = Text.encodeUtf8("root page :" # path);
                      headers = [];
                      status_code = 200;
                      streaming_strategy = null;
                  };
              };

spencer · October 6, 2021, 11:26pm

@kevinli
You can use the code I’m working on here:

I’m working on something similar to you, trying to make a kind of google storage bucket, but that can also be used for direct links to videos/images/any type of file. Hope that helps

peterparker · October 7, 2021, 11:06am

I was literally trying to generate a random secret token as in firebase storage and saw you post. Your random fileid generator looks super interesting, thanks for the share!

spencer · October 9, 2021, 11:48pm

Credit goes to @Gabriel !

paulyoung · January 23, 2022, 1:34am

@cryptoschindler I think it’s here:

github.com

dfinity/agent-rs/blob/a8bb38e2079c935fc4deec0128c05319a43e5551/ic-asset/src/sync.rs#L16


      
          use crate::operations::{
              create_new_assets, delete_obsolete_assets, set_encodings, unset_obsolete_encodings,
          };
          use crate::plumbing::{make_project_assets, AssetLocation, ProjectAsset};
          use ic_utils::Canister;
          use std::collections::HashMap;
          use std::path::Path;
          use std::time::Duration;
          use walkdir::WalkDir;
          
          pub async fn sync(canister: &Canister<'_>, dir: &Path, timeout: Duration) -> anyhow::Result<()> {
              let asset_locations = gather_asset_locations(dir);
          
              let canister_call_params = CanisterCallParams { canister, timeout };
          
              let container_assets = list_assets(&canister_call_params).await?;
          
              println!("Starting batch.");
          
              let batch_id = create_batch(&canister_call_params).await?;

heyuanxun · January 24, 2022, 3:58am

I want to return some chunk information through https://deqv7-6qaaa-aaaai-qfkna-cai.raw.ic0.app/ .
But as you see the Callback did not work.
It just return the first chunk.
So,how can i return information thurough Callback to return chunk information.
This my code http_demo/main.mo at main · xiaoyuanxun/http_demo · GitHub
Hope someone would help me
I will be very grateful.

cyberowl · February 9, 2022, 5:26am

What has been the performance on that approach for say fetching 200 images?

heyuanxun · February 10, 2022, 9:02am

Because query call response limit 3M.So it may be to use a little time

cyberowl · February 10, 2022, 9:07am

Yeah seems like it should be fast. Going to be testing soon and see how it goes.

Topic		Replies	Views
Question on http_request() Developers	17	3193	August 28, 2021
Serve Templated HTML from Canister via HTTP Programs & Applications	4	409	May 19, 2024
Custom asset canister does not provide files as raw Developers	17	906	November 29, 2023
How do some canisters handle http requests Developers	2	1999	January 24, 2022
Server side rendering? (Or a way around it?) Developers	1	408	September 27, 2021

Exposing URLs for querying images

Related topics