Short Update on My Search Problem
I’ve developed a prototype for a German-language stemmer with Motoko. While I’m not certain that all Porter stemming rules are fully implemented, the core stemming functionality is working. The parser also excludes a predefined list of stop words and effectively removes all HTML tags.
However, the initial indexing process is currently very slow and takes a significant amount of time. Additionally, many unusable words are being indexed, and some words that appear frequently across issues cause the specific word arrays to grow excessively large.
In theory, both the indexing and search functionalities are working, but I’m unsure how to scale this approach effectively with my dataset of 22,000 issues and 90,000 messages and more.