Bug 36678 - Include fields with non-filing characters removed when indexing
Summary: Include fields with non-filing characters removed when indexing
Status: Pushed to main
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: unspecified
Hardware: All All
: P5 - low normal (vote)
Assignee: Nick Clemens (kidclamp)
QA Contact: Martin Renvoize
URL:
Keywords: release-notes-needed
Depends on:
Blocks:
 
Reported: 2024-04-23 18:57 UTC by Nick Clemens (kidclamp)
Modified: 2024-05-02 11:20 UTC (History)
4 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
24.05.00


Attachments
Sample record for testing (3.86 KB, application/xml)
2024-04-23 18:57 UTC, Nick Clemens (kidclamp)
Details
Bug 36678: Adjust unit tests (1.64 KB, patch)
2024-04-23 19:04 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 36678: Index fields with non-filing characters in both versions (2.60 KB, patch)
2024-04-23 19:04 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 36678: Adjust unit tests (1.69 KB, patch)
2024-04-26 10:32 UTC, David Nind
Details | Diff | Splinter Review
Bug 36678: Index fields with non-filing characters in both versions (2.65 KB, patch)
2024-04-26 10:32 UTC, David Nind
Details | Diff | Splinter Review
Bug 36678: Adjust unit tests (1.75 KB, patch)
2024-05-01 12:11 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 36678: Index fields with non-filing characters in both versions (2.71 KB, patch)
2024-05-01 12:11 UTC, Martin Renvoize
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Nick Clemens (kidclamp) 2024-04-23 18:57:15 UTC
Created attachment 165438 [details]
Sample record for testing

Currently we only remove non-filing characters for sort fields, however, this can make searching difficult.

Consider:
L'amour de l'art : art contemporain et collections privées du Sud-Ouest. 

Currently a search for:
amour de l'art

will return no results in elastic search

We should include the form with non-filing characters removed

See also bug 21357
Comment 1 Nick Clemens (kidclamp) 2024-04-23 19:04:11 UTC
Created attachment 165439 [details] [review]
Bug 36678: Adjust unit tests
Comment 2 Nick Clemens (kidclamp) 2024-04-23 19:04:15 UTC
Created attachment 165440 [details] [review]
Bug 36678: Index fields with non-filing characters in both versions

Currently we only remove non-filing characters for sort fields, however, this can make searching difficult.
This patch adds the filing form to the index as well to aid in searching.

To test:
 0 - Setup Koha with Elasticsearch
 1 - Import the sample record on this report: "L'amour de l'art"
 2 - Search for "amour de l'art" - no results
 3 - Apply patch
 4 - Reindex
 5 - Search for "amour de l'art" - result!
 6 - Search for "title:amour de l'art" - result!
Comment 3 David Nind 2024-04-26 10:32:42 UTC
Created attachment 165648 [details] [review]
Bug 36678: Adjust unit tests

Signed-off-by: David Nind <david@davidnind.com>
Comment 4 David Nind 2024-04-26 10:32:45 UTC
Created attachment 165649 [details] [review]
Bug 36678: Index fields with non-filing characters in both versions

Currently we only remove non-filing characters for sort fields, however, this can make searching difficult.
This patch adds the filing form to the index as well to aid in searching.

To test:
 0 - Setup Koha with Elasticsearch
 1 - Import the sample record on this report: "L'amour de l'art"
 2 - Search for "amour de l'art" - no results
 3 - Apply patch
 4 - Reindex
 5 - Search for "amour de l'art" - result!
 6 - Search for "title:amour de l'art" - result!

Signed-off-by: David Nind <david@davidnind.com>
Comment 5 Martin Renvoize 2024-05-01 12:11:44 UTC
Created attachment 165960 [details] [review]
Bug 36678: Adjust unit tests

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 6 Martin Renvoize 2024-05-01 12:11:47 UTC
Created attachment 165961 [details] [review]
Bug 36678: Index fields with non-filing characters in both versions

Currently we only remove non-filing characters for sort fields, however, this can make searching difficult.
This patch adds the filing form to the index as well to aid in searching.

To test:
 0 - Setup Koha with Elasticsearch
 1 - Import the sample record on this report: "L'amour de l'art"
 2 - Search for "amour de l'art" - no results
 3 - Apply patch
 4 - Reindex
 5 - Search for "amour de l'art" - result!
 6 - Search for "title:amour de l'art" - result!

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 7 Martin Renvoize 2024-05-01 12:12:30 UTC
Nice improvement, covered by tests, works as described and QA scripts happy.

Passing QA
Comment 8 Katrin Fischer 2024-05-02 11:20:09 UTC
Pushed for 24.05!

Well done everyone, thank you!