1
1
mirror of https://github.com/go-gitea/gitea synced 2025-07-23 02:38:35 +00:00

Updated tokenizer to better matching when search for code snippets (#32261)

This PR improves the accuracy of Gitea's code search. 

Currently, Gitea does not consider statements such as
`onsole.log("hello")` as hits when the user searches for `log`. The
culprit is how both ES and Bleve are tokenizing the file contents (in
both cases, `console.log` is a whole token).

In ES' case, we changed the tokenizer to
[simple_pattern_split](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-simplepatternsplit-tokenizer.html#:~:text=The%20simple_pattern_split%20tokenizer%20uses%20a,the%20tokenization%20is%20generally%20faster.).
In such a case, tokens are words formed by digits and letters. In
Bleve's case, it employs a
[letter](https://blevesearch.com/docs/Tokenizers/) tokenizer.

Resolves #32220

---------

Signed-off-by: Bruno Sofiato <bruno.sofiato@gmail.com>
This commit is contained in:
Bruno Sofiato
2024-11-06 17:51:20 -03:00
committed by GitHub
parent b573512312
commit f64fbd9b74
17 changed files with 83 additions and 12 deletions

View File

@@ -4,5 +4,6 @@ This repository will be used to test code search. The snippet below shows its di
├── avocado.md
├── cucumber.md
├── ham.md
── potato
└── ham.md
── potato
| └── ham.md
└── example-file.js