Bug 32937 - Zebra: Ignore copyright symbol when searching
Summary: Zebra: Ignore copyright symbol when searching
Status: RESOLVED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Zebra (show other bugs)
Version: Main
Hardware: All All
: P5 - low minor (vote)
Assignee: Katrin Fischer
QA Contact:
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-02-10 12:05 UTC by ashley.day
Modified: 2023-07-25 00:03 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
23.05.00,22.11.06,22.05.14


Attachments
Bug 32937: Make Zebra ignore the copyright symbol in searches (1.87 KB, patch)
2023-04-16 12:02 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 32937: Make Zebra ignore the copyright symbol in searches (1.92 KB, patch)
2023-04-16 23:07 UTC, David Nind
Details | Diff | Splinter Review
Bug 32937: Make Zebra ignore the copyright symbol in searches (1.97 KB, patch)
2023-04-28 17:18 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description ashley.day 2023-02-10 12:05:09 UTC
Our library includes the copyright symbol when cataloguing the publication date (264$c) according to RDA 2.11.1.3. 

Searching dates as free text (not using the advanced date search) is an issue for us in both the opac and staff interface. Koha is not pulling all possible results because we are using the copyright symbol in the record. I've been advised that Koha treats ©2023 and 2023 as two different words.

Of course when adding the copyright symbol to the date to the general search, the correct number of results are shown. 

We don't want to require a user to know alt codes to produce a copyright symbol, and many students simply are not going to use an advanced date search.

Is it possible for KOHA to have the ability to ignore a copyright symbol in 264c for free text searching and return search results based on the date alone?
Comment 1 Katrin Fischer 2023-04-16 11:56:56 UTC
For searching you should catalog the normalized date in 008 7-10, this will be used for date range search and might improve things a bit.

I will also attach patches that work for ignoring © with Zebra. I am not sure if a change is needed for Elasticsearch (cannot test it currently) and I would not know where to make the changes. If you are using Elastisearch and can confirm this is a problem there too, would you mind filing another bug for the "Searching - Elasticsearch" component?
Comment 2 Katrin Fischer 2023-04-16 12:02:53 UTC
Created attachment 149728 [details] [review]
Bug 32937: Make Zebra ignore the copyright symbol in searches

It's common to catalog the year with the copyright symbol attached
to it. This creates some issues with search that we can fix by
adding the © to the list of characters Zebra ignores in search.

To test:
* Search for any existing publication year with ©, example: ©1951
* Verify your record is not found
* Add the copyright symbol to a record, verify it's now found with ©,
  but not without
* Apply patch
* On ktd:
  * sudo cp -i /kohadevbox/koha/etc/zebradb/etc/word-phrase-utf.chr  /etc/koha/zebradb/etc/word-phrase-utf.chr
  * sudo koha-zebra --restart kohadev
  * sudo koha-mysql kohadev
  * DELETE FROM biblio WHERE biblionumber = 369;
  * The reindex woudl fail for me with this broken record present.
  * sudo koha-rebuild-zebra -f kohadev
* Repeat searches, the records should be found when searching
  with and without ©.
Comment 3 David Nind 2023-04-16 23:07:40 UTC
Created attachment 149736 [details] [review]
Bug 32937: Make Zebra ignore the copyright symbol in searches

It's common to catalog the year with the copyright symbol attached
to it. This creates some issues with search that we can fix by
adding the © to the list of characters Zebra ignores in search.

To test:
* Search for any existing publication year with ©, example: ©1951
* Verify your record is not found
* Add the copyright symbol to a record, verify it's now found with ©,
  but not without
* Apply patch
* On ktd:
  * sudo cp -i /kohadevbox/koha/etc/zebradb/etc/word-phrase-utf.chr  /etc/koha/zebradb/etc/word-phrase-utf.chr
  * sudo koha-zebra --restart kohadev
  * sudo koha-mysql kohadev
  * DELETE FROM biblio WHERE biblionumber = 369;
  * The reindex woudl fail for me with this broken record present.
  * sudo koha-rebuild-zebra -f kohadev
* Repeat searches, the records should be found when searching
  with and without ©.

Signed-off-by: David Nind <david@davidnind.com>
Comment 4 David Nind 2023-04-17 00:00:58 UTC
(In reply to Katrin Fischer from comment #1)

> I will also attach patches that work for ignoring © with Zebra. I am not
> sure if a change is needed for Elasticsearch (cannot test it currently) and
> I would not know where to make the changes. If you are using Elastisearch
> and can confirm this is a problem there too, would you mind filing another
> bug for the "Searching - Elasticsearch" component?

I tested with KTD and Elasticsearch and OpenSearch (ktd --es7 up and ktd --os1 up). The record was found with and without the copyright symbol, so I don't think it affects Koha with Elasticsearch or OpenSearch.

Note: I couldn't get KTD to start properly using --es6, --es8, and --os2.

Testing notes:
1. Edited the BKS framework so that 264$c is visible in the editor.
2. Add a year to 264$c with and without the copyright symbol, as per the test plan.
Comment 5 Nick Clemens (kidclamp) 2023-04-28 17:18:59 UTC
Created attachment 150404 [details] [review]
Bug 32937: Make Zebra ignore the copyright symbol in searches

It's common to catalog the year with the copyright symbol attached
to it. This creates some issues with search that we can fix by
adding the © to the list of characters Zebra ignores in search.

To test:
* Search for any existing publication year with ©, example: ©1951
* Verify your record is not found
* Add the copyright symbol to a record, verify it's now found with ©,
  but not without
* Apply patch
* On ktd:
  * sudo cp -i /kohadevbox/koha/etc/zebradb/etc/word-phrase-utf.chr  /etc/koha/zebradb/etc/word-phrase-utf.chr
  * sudo koha-zebra --restart kohadev
  * sudo koha-mysql kohadev
  * DELETE FROM biblio WHERE biblionumber = 369;
  * The reindex woudl fail for me with this broken record present.
  * sudo koha-rebuild-zebra -f kohadev
* Repeat searches, the records should be found when searching
  with and without ©.

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 6 Tomás Cohen Arazi 2023-05-12 14:20:56 UTC
Pushed to master for 23.05.

Nice work everyone, thanks!
Comment 7 Matt Blenkinsop 2023-05-15 12:42:25 UTC
Nice work everyone!

Pushed to stable for 22.11.x
Comment 8 Lucas Gass 2023-06-15 19:16:15 UTC
Backported to 22.05.x for upcoming 22.05.14
Comment 9 Danyon Sewell 2023-07-25 00:02:59 UTC
Not backporting to 21.11.x