How can I import and parse a local csv file (i.e., part of the project) in Motoko?

Developers Motoko
1

As a learning exercise I’m attempting to recreate a small Rust program in Motoko. It takes a local csv file as input.

The csv file looks like this (rows and columns truncated for brevity)…

label,pixel0,pixel1,pixel2
1,0,0,0
0,0,0,0
1,0,0,0
4,0,0,0

The Rust program contains code like this where path is the name of the file .

let file = File::open(path).unwrap_or_else(|err| {
    eprintln!("Problem opening file: {path} - {err}");
    process::exit(1);
});
let reader = BufReader::new(file);
// Skip header
let rows = reader.lines().skip(1);

for row in rows 
{
    let r = row.unwrap_or_else(|err| {
        eprintln!("Problem extracting observation row from input: {err} in file: {path}");
        process::exit(1);
    });

    let comma_separated: Vec<&str> = r.split(',').collect();
    let label = comma_separated[0];

    let pixel_strings = &comma_separated[1..];
    let mut pixels:  Vec<i32> = Vec::new();

    for pixel_string in pixel_strings {
        let pixel: i32 = pixel_string.parse().unwrap_or_else(|err| {
            eprintln!("Problem converting pixel string into integer: {err} in file: {path}");
            process::exit(1);
        });
        pixels.push(pixel);
    }
}

So in the above

First row is "1,0,0,0"
Second row is "0,0,0,0"

etc

First pixels vector is [1, 0, 0, 0]
Second pixels vector is [0, 0, 0, 0]

etc

I’ve been trying unsuccessfully to recreate this in Motoko with the assistance of a few genAIs. I gather that I need to place my csv file in an assets folder. However, the code they’ve generated so far always falls over somewhere.

This was the initial AI attempt that was served up to me.

import Text "mo:base/Text";
import Array "mo:base/Array";

actor Main {
    public func readCSV() : async () {
        // Read the entire file content
        let fileContent = await Text.readFile("data.csv");

        // Split the content into lines
        let lines = Text.split(fileContent, #char '\n');

        // Skip the header
        let dataLines = Array.slice(lines, 1, lines.size());

        for (line in dataLines.vals()) {
            if (line != "") {
                // Split each line by commas
                let values = Text.split(line, #char ',');

                // Extract the label
                let label = values[0];

                // Process pixel values
                let pixels = Array.map(values.slice(1), func(v) {
                    switch (Text.toNat(v)) {
                        case (?natValue) natValue;
                        case null {
                            Debug.trap("Invalid pixel value");
                        };
                    }
                });

                // Here you can add your logic to process the label and pixels
                Debug.print("Label: " # debug_show(label));
                Debug.print("Pixels: " # debug_show(pixels));
            }
        };
    };

    public func start() {
        readCSV();
    }
}

From elsewhere, it seems that there was no chance the above would work and that I need to make use of Http requests and blob transformations but, so far I’ve been unable to come up with something that compiles.

2

Your Rust program can access the file locally, but your Motoko code won’t be able to find it unless you’ve uploaded it to your canister. To solve this, you need to write Motoko code for your actor that allows you to upload the file. Then you’ll be able to parse it it. Also several functions used in your in your initial AI attempt does not exist, like Text.readFile for instance.

Here’s an example for uploading an UTF-8 encoded CSV file: https://m7sm4-2iaaa-aaaab-qabra-cai.raw.ic0.app/?tag=161353968 (Haven’t tested it so keep that in mind)

Once your file is uploaded in order parse the content, you can use the Text and Array libraries in Motoko.

There are more advanced parsing libraries available in Motoko, but for your case I’m sure you can get it done with the Base library.

Good luck

1 Like
3

How do you get the file into that uploadChunk() to start with?

4

You’d have to write a script that reads from your local file and turn it into chunks.