mirror of
https://github.com/go-gitea/gitea
synced 2025-07-22 18:28:37 +00:00
Updated tokenizer to better matching when search for code snippets (#32261)
This PR improves the accuracy of Gitea's code search. Currently, Gitea does not consider statements such as `onsole.log("hello")` as hits when the user searches for `log`. The culprit is how both ES and Bleve are tokenizing the file contents (in both cases, `console.log` is a whole token). In ES' case, we changed the tokenizer to [simple_pattern_split](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-simplepatternsplit-tokenizer.html#:~:text=The%20simple_pattern_split%20tokenizer%20uses%20a,the%20tokenization%20is%20generally%20faster.). In such a case, tokens are words formed by digits and letters. In Bleve's case, it employs a [letter](https://blevesearch.com/docs/Tokenizers/) tokenizer. Resolves #32220 --------- Signed-off-by: Bruno Sofiato <bruno.sofiato@gmail.com>
This commit is contained in:
@@ -4,5 +4,6 @@ This repository will be used to test code search. The snippet below shows its di
|
||||
├── avocado.md
|
||||
├── cucumber.md
|
||||
├── ham.md
|
||||
└── potato
|
||||
└── ham.md
|
||||
├── potato
|
||||
| └── ham.md
|
||||
└── example-file.js
|
@@ -3,7 +3,7 @@
|
||||
65f1bf27bc3bf70f64657658635e66094edbcb4d refs/heads/develop
|
||||
65f1bf27bc3bf70f64657658635e66094edbcb4d refs/heads/feature/1
|
||||
78fb907e3a3309eae4fe8fef030874cebbf1cd5e refs/heads/home-md-img-check
|
||||
3731fe53b763859aaf83e703ee731f6b9447ff1e refs/heads/master
|
||||
9f894b61946fd2f7b8b9d8e370e4d62f915522f5 refs/heads/master
|
||||
62fb502a7172d4453f0322a2cc85bddffa57f07a refs/heads/pr-to-update
|
||||
4649299398e4d39a5c09eb4f534df6f1e1eb87cc refs/heads/sub-home-md-img-check
|
||||
3fa2f829675543ecfc16b2891aebe8bf0608a8f4 refs/notes/commits
|
||||
|
Binary file not shown.
@@ -1,2 +1,2 @@
|
||||
P pack-393dc29256bc27cb2ec73898507df710be7a3cf5.pack
|
||||
P pack-a7bef76cf6e2b46bc816936ab69306fb10aea571.pack
|
||||
|
||||
|
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -4,7 +4,7 @@
|
||||
65f1bf27bc3bf70f64657658635e66094edbcb4d refs/heads/develop
|
||||
65f1bf27bc3bf70f64657658635e66094edbcb4d refs/heads/feature/1
|
||||
78fb907e3a3309eae4fe8fef030874cebbf1cd5e refs/heads/home-md-img-check
|
||||
3731fe53b763859aaf83e703ee731f6b9447ff1e refs/heads/master
|
||||
9f894b61946fd2f7b8b9d8e370e4d62f915522f5 refs/heads/master
|
||||
62fb502a7172d4453f0322a2cc85bddffa57f07a refs/heads/pr-to-update
|
||||
4649299398e4d39a5c09eb4f534df6f1e1eb87cc refs/heads/sub-home-md-img-check
|
||||
3fa2f829675543ecfc16b2891aebe8bf0608a8f4 refs/notes/commits
|
||||
|
Reference in New Issue
Block a user