Bug 36678

Summary: Include fields with non-filing characters removed when indexing
Product: Koha Reporter: Nick Clemens (kidclamp) <nick>
Component: Searching - ElasticsearchAssignee: Nick Clemens (kidclamp) <nick>
Status: Pushed to main --- QA Contact: Martin Renvoize <martin.renvoize>
Severity: normal    
Priority: P5 - low CC: dcook, januszop, m.de.rooy, martin.renvoize
Version: unspecifiedKeywords: release-notes-needed
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21357
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
24.05.00
Attachments: Sample record for testing
Bug 36678: Adjust unit tests
Bug 36678: Index fields with non-filing characters in both versions
Bug 36678: Adjust unit tests
Bug 36678: Index fields with non-filing characters in both versions
Bug 36678: Adjust unit tests
Bug 36678: Index fields with non-filing characters in both versions

Description Nick Clemens (kidclamp) 2024-04-23 18:57:15 UTC
Created attachment 165438 [details]
Sample record for testing

Currently we only remove non-filing characters for sort fields, however, this can make searching difficult.

Consider:
L'amour de l'art : art contemporain et collections privées du Sud-Ouest. 

Currently a search for:
amour de l'art

will return no results in elastic search

We should include the form with non-filing characters removed

See also bug 21357
Comment 1 Nick Clemens (kidclamp) 2024-04-23 19:04:11 UTC
Created attachment 165439 [details] [review]
Bug 36678: Adjust unit tests
Comment 2 Nick Clemens (kidclamp) 2024-04-23 19:04:15 UTC
Created attachment 165440 [details] [review]
Bug 36678: Index fields with non-filing characters in both versions

Currently we only remove non-filing characters for sort fields, however, this can make searching difficult.
This patch adds the filing form to the index as well to aid in searching.

To test:
 0 - Setup Koha with Elasticsearch
 1 - Import the sample record on this report: "L'amour de l'art"
 2 - Search for "amour de l'art" - no results
 3 - Apply patch
 4 - Reindex
 5 - Search for "amour de l'art" - result!
 6 - Search for "title:amour de l'art" - result!
Comment 3 David Nind 2024-04-26 10:32:42 UTC
Created attachment 165648 [details] [review]
Bug 36678: Adjust unit tests

Signed-off-by: David Nind <david@davidnind.com>
Comment 4 David Nind 2024-04-26 10:32:45 UTC
Created attachment 165649 [details] [review]
Bug 36678: Index fields with non-filing characters in both versions

Currently we only remove non-filing characters for sort fields, however, this can make searching difficult.
This patch adds the filing form to the index as well to aid in searching.

To test:
 0 - Setup Koha with Elasticsearch
 1 - Import the sample record on this report: "L'amour de l'art"
 2 - Search for "amour de l'art" - no results
 3 - Apply patch
 4 - Reindex
 5 - Search for "amour de l'art" - result!
 6 - Search for "title:amour de l'art" - result!

Signed-off-by: David Nind <david@davidnind.com>
Comment 5 Martin Renvoize 2024-05-01 12:11:44 UTC
Created attachment 165960 [details] [review]
Bug 36678: Adjust unit tests

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 6 Martin Renvoize 2024-05-01 12:11:47 UTC
Created attachment 165961 [details] [review]
Bug 36678: Index fields with non-filing characters in both versions

Currently we only remove non-filing characters for sort fields, however, this can make searching difficult.
This patch adds the filing form to the index as well to aid in searching.

To test:
 0 - Setup Koha with Elasticsearch
 1 - Import the sample record on this report: "L'amour de l'art"
 2 - Search for "amour de l'art" - no results
 3 - Apply patch
 4 - Reindex
 5 - Search for "amour de l'art" - result!
 6 - Search for "title:amour de l'art" - result!

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 7 Martin Renvoize 2024-05-01 12:12:30 UTC
Nice improvement, covered by tests, works as described and QA scripts happy.

Passing QA
Comment 8 Katrin Fischer 2024-05-02 11:20:09 UTC
Pushed for 24.05!

Well done everyone, thank you!