Skip to content

Excuse me, how to support other language search, such as Chinese search, thank you #201

Open

Description

@sinianzhiren

Excuse me, how to support other language search, such as Chinese search, thank you .

Activity

  1. lucaong commented on Feb 1, 2023

    @lucaong
    Owner

    Hello @sinianzhiren ,
    in principle, MiniSearch should be able to work with any language. In practice, in some cases it might be necessary to tweak some options, but I think that the defaults should be a good starting point.

    Unfortunately I do not know enough about Chinese language to guide you here, but other users have successfully used MiniSearch for Chinese (look for example at this comment or at this issue).

    Did you encounter a specific problem supporting Chinese or other languages? If so, you can describe it, and I would be happy to help if I can.

  2. SSShooter commented on Feb 7, 2023

    @SSShooter

    Excuse me, how to support other language search, such as Chinese search, thank you .

    You should do Chinese word segmentation with library like nodejieba before indexing documents.

  3. connebs commented on Jul 4, 2023

    @connebs

    If you don't care about supporting Firefox, Intl.Segmenter is great.

  4. mantou132 commented on Jan 14, 2024

    @mantou132

    Firefox Nightly support Intl.Segmenter.

    Use Intl.Segmenter support CJK example:

    const segmenter =
      Intl.Segmenter && new Intl.Segmenter("zh", { granularity: "word" });
    
    const miniSearch = new MiniSearch({
      fields: ["text"],
      processTerm: (term) => {
        if (!segmenter) return term;
        const tokens = [];
        for (const seg of segmenter.segment(term)) {
          tokens.push(seg.segment);
        }
        return tokens;
      },
    });
    
    const documents = [
      { id: 1, text: "为字段添加 required 属性,并在提交时进行表单验证" },
      {
        id: 2,
        text: "By default, the same processing is applied to search queries. In order to apply a different processing to search queries, supply a processTerm search option:",
      },
    ];
    
    miniSearch.addAll(documents);
    console.log("===");
    console.log(miniSearch.search("添加"));

    This is online example: https://duoyun-ui.gemjs.org/zh/
    Search front end use @docsearch/js

  5. ThomasChan commented on Jul 15, 2024

    @ThomasChan

    I also encountered the problem of searching Chinese, for example, when searching for "预置", due to the problem of word segmentation, the content cannot be searched due to the word segmentation of "预" and "置", my project uses vitepress, indirectly uses minisearch, and finally I configured it like this to support search:

    ...
    export default defineConfig({
      ...
      themeConfig: {
        search: {
          options: {
            miniSearch: {
              options: {
                tokenize: (term) => {
                  if (typeof term === 'string') term = term.toLowerCase();
                  // @ts-ignore
                  const segmenter = Intl.Segmenter && new Intl.Segmenter("zh", { granularity: "word" });
                  if (!segmenter) return [term];
                  const tokens = [];
                  for (const seg of segmenter.segment(term)) {
                    // @ts-ignore
                    tokens.push(seg.segment);
                  }
                  return tokens;
                },
              },
              searchOptions: {
                combineWith: 'AND', // important for search chinese
                processTerm: (term) => {
                  if (typeof term === 'string') term = term.toLowerCase();
                  // @ts-ignore
                  const segmenter = Intl.Segmenter && new Intl.Segmenter("zh", { granularity: "word" });
                  if (!segmenter) return term;
                  const tokens = [];
                  for (const seg of segmenter.segment(term)) {
                    // @ts-ignore
                    tokens.push(seg.segment);
                  }
                  return tokens;
                },
              },
            },
          },
        },
      },
      ...
    });

    Thanks to @mantou132

  6. onyxblade commented on Nov 30, 2024

    @onyxblade

    I found that Intl.Segmenter is not reliable enough for search. For example, '懵逼了' in a document is broken down ['懵', '逼了'], while a search term '懵逼' is broken down ['懵', '逼']. This damages searchability.

    I use bigram for Chinese search instead. Since minisearch can support different fuzzy option based on term, I can disable fuzzy search for Chinese terms and let it use bigram, and still have fuzzy search enabled for English terms. This works so far well for my usecase.

    import { bigram } from "n-gram"
    
    const SPACE_OR_PUNCTUATION = /[\n\r\p{Z}\p{P}]+/u
    // From https://github.com/vinta/pangu.js/blob/master/src/shared/core.js
    const CJK_RANGE = '\u2e80-\u2eff\u2f00-\u2fdf\u3040-\u309f\u30a0-\u30fa\u30fc-\u30ff\u3100-\u312f\u3200-\u32ff\u3400-\u4dbf\u4e00-\u9fff\uf900-\ufaff'
    const CJK_NCJK = new RegExp(`([${CJK_RANGE}])([^${CJK_RANGE}])`, 'g')
    const NCJK_CJK = new RegExp(`([^${CJK_RANGE}])([${CJK_RANGE}])`, 'g')
    const CJK_WORD = new RegExp(`^[${CJK_RANGE}]+$`)
    
    function isCJKTerm(term: string) {
      return !!term.match(CJK_WORD)
    }
    
    // Function to add space between CJK and non-CJK characters.
    // '中文Latin中文' => '中文 Latin 中文'
    function addSpaceBetweenCJKandNonCJK(text: string) {
      return text.replace(CJK_NCJK, '$1 $2').replace(NCJK_CJK, '$1 $2')
    }
    
    const miniSearch = new MiniSearch({
      tokenize(text) {
        const tokens: string[] = []
    
        // Add space between CJK and non-CJK, and then split them by empty space.
        const segments = addSpaceBetweenCJKandNonCJK(text).split(SPACE_OR_PUNCTUATION)
    
        segments.forEach(segment => {
          if (isCJKTerm(segment)) {
            // Conversion between Tradictional Chinese and Simplified Chinese can happen here.
            // A simple character table can be found at:
            // https://github.com/tongwentang/tongwen-dict/blob/main/src/charater/t2s-char.json
    
            // Each single character is added. '樣例詞組' => ['樣', '例', '詞', '組']
            Array.from(segment).forEach(char => tokens.push(char))
            // Each bigram is added. '樣例詞組' => ['樣例', '例詞', '詞組']
            bigram(segment).forEach(token => tokens.push(token))
          } else {
            // For non-CJK terms, directly add it to tokens.
            tokens.push(segment)
          }
        })
        return tokens
      },
      searchOptions: {
        combineWith: 'AND',
        fuzzy(term) {
          // For CJK terms, disable fuzzy search. Otherwise, use a fuzzy option.
          if (isCJKTerm(term)) {
            return false
          } else {
            return 0.35
          }
        },
        maxFuzzy: 4
      }
    })
  7. added a commit that references this issue on Dec 30, 2024
  8. Sun-ZhenXing commented on Feb 1, 2025

    @Sun-ZhenXing

    As the comment above says, Intl.Segmenter actually works poorly.

    This is my approach, if you need to use jieba both in Browser & Node.js, I recommend using jieba-wasm which is built on Rust and WebAssembly.

    Here is the vitepress configuration:

    // config.ts
    import { cut_for_search } from 'jieba-wasm'
    
    export default {
      themeConfig: {
        search: {
          provider: 'local',
          options: {
            miniSearch: {
              options: {
                tokenize,
              },
              searchOptions: {
                combineWith: 'AND',
                processTerm: tokenize,
              }
            },
          },
        },
      },
    }
    
    function tokenize(term: string): string[] {
      if (typeof term !== 'string') {
        return []
      }
      const cut = globalThis.cut_for_search || cut_for_search
      return cut(term)
        .map((word: string) => word.trim().toLocaleLowerCase())
        .filter((word: string) => word.length >= 1)
    }
    // theme/index.ts
    import DefaultTheme from 'vitepress/theme'
    import init, { cut_for_search } from 'jieba-wasm'
    
    export default {
      extends: DefaultTheme,
      async enhanceApp({ app }) {
        if (!import.meta.env.SSR) {
          // Load the jieba-wasm module only in the browser
          await init()
          globalThis.cut_for_search = cut_for_search
        }
      }
    }

    Now it works fine in my document, but opening a page requires pre-loading a ~3MB WASM file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions