Bug 31471 - Duplicate check in cataloguing fails with Elasticsearch for records with multiple ISBN
Summary: Duplicate check in cataloguing fails with Elasticsearch for records with mult...
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: Nick Clemens
QA Contact:
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-08-26 15:57 UTC by Katrin Fischer
Modified: 2023-12-28 20:42 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
23.05.00,22.11.04,22.05.11,21.11.19


Attachments
Bug 31471: Send multiple ISBN with 'OR' for FindDuplicate (3.62 KB, patch)
2023-02-24 14:09 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 31471: Send multiple ISBN with 'OR' for FindDuplicate (3.68 KB, patch)
2023-03-01 07:03 UTC, Biblibre Sandboxes
Details | Diff | Splinter Review
Bug 31471: Send multiple ISBN with 'OR' for FindDuplicate (3.73 KB, patch)
2023-03-04 13:32 UTC, Katrin Fischer
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Katrin Fischer 2022-08-26 15:57:10 UTC
The duplicate check in cataloguing doesn't work with Elasticsearch.


To test:
* With Zebra: Download a record via Z39.50 and save.
* Make sure the record is found in the catalog.
* Download the same record via Z39.50 again and try to save - duplicate confirm message pops up.
* Switch to Zebra
* Again, make sure that the record can be found in the catalog.
* Download the record again via Z39.50 and try to save - no duplicate warning, the record is saved.

If an example is needed we could provide a taget and sample.

Note: As I understand FindDuplicate it checks for ISSN and after that a combination of title and author. Is there a thing we might be missing here?

Tested in different installations with master and 21.11.
Comment 1 Katrin Fischer 2022-09-20 08:37:14 UTC
Ok, I have found the issue:

The problem appears for records with multiple ISBN.

An example:
* Make sure you are using Elasticsearch
* Z39.50 target: LOC 
* Search for: Control number: 14455023
* Import and save
* Search for the record again
* Import and save - duplicate check hasn't been triggered

Taking a closer look:
The duplicate check using FindDuplicate will create the search: 

isbn:1600213510 | 9781600213519

If you search this in the catalog search, it will yield no results with Elasticsearch, but it will indeed work with Zebra, I guess because the | is treated differently in this case.
Comment 2 Katrin Fischer 2022-09-20 08:57:04 UTC
I am not sure how the search for Elasticsearch should be built in order to work reliably, help and hints appreciated!
Comment 3 Nick Clemens 2023-02-24 14:09:47 UTC
Created attachment 147333 [details] [review]
Bug 31471: Send multiple ISBN with 'OR' for FindDuplicate

When a record has multiple ISBNs the database stores them seperated by a
pipe

Zebra interprets a string like "isbn:1600213510 | 9781600213519" as an 'OR' search, but for
Elasticsearch we need to explicitly send "OR" - and Zebra support this
as well

To test:
1 - Make sure you are using Elasticsearch
2 - Cataloging->Add a new record from Z3950
3 - Choose target: LOC
4 - Search for: Control number: 14455023
5 - Import and save
6 - Search for the record again
7 - Import and save - duplicate check isn't been triggered
8 - Apply patch
9 - restart_all
10 - repeat 2-7
11 - this time you should get a duplicate notification
12 - Sitch SearchEngine syspref to 'Zebra'
13 - repeat 2-7
14 - Ensure you still get duplicate notification
Comment 4 Biblibre Sandboxes 2023-03-01 07:03:25 UTC
Created attachment 147530 [details] [review]
Bug 31471: Send multiple ISBN with 'OR' for FindDuplicate

When a record has multiple ISBNs the database stores them seperated by a
pipe

Zebra interprets a string like "isbn:1600213510 | 9781600213519" as an 'OR' search, but for
Elasticsearch we need to explicitly send "OR" - and Zebra support this
as well

To test:
1 - Make sure you are using Elasticsearch
2 - Cataloging->Add a new record from Z3950
3 - Choose target: LOC
4 - Search for: Control number: 14455023
5 - Import and save
6 - Search for the record again
7 - Import and save - duplicate check isn't been triggered
8 - Apply patch
9 - restart_all
10 - repeat 2-7
11 - this time you should get a duplicate notification
12 - Sitch SearchEngine syspref to 'Zebra'
13 - repeat 2-7
14 - Ensure you still get duplicate notification

Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Comment 5 Michaela Sieber 2023-03-01 07:04:56 UTC
Tested with LoC Control No.


1.)	16509983 (100 countries, 5000 ideas), ISBN: 9781426207587 (softcover : alk. paper) :; 1426207581 (softcover : alk. paper)
and  
2.)	19483458 (Cyberdoc), ISBN: 9780985248314 (cloth); 9780985248321 (pbk.)


Thanks for the patch!
Comment 6 Katrin Fischer 2023-03-04 13:22:15 UTC
The solution can be so easy, once you see it :) Thank you, Nick!
Comment 7 Katrin Fischer 2023-03-04 13:32:48 UTC
Created attachment 147752 [details] [review]
Bug 31471: Send multiple ISBN with 'OR' for FindDuplicate

When a record has multiple ISBNs the database stores them seperated by a
pipe

Zebra interprets a string like "isbn:1600213510 | 9781600213519" as an 'OR' search, but for
Elasticsearch we need to explicitly send "OR" - and Zebra support this
as well

To test:
1 - Make sure you are using Elasticsearch
2 - Cataloging->Add a new record from Z3950
3 - Choose target: LOC
4 - Search for: Control number: 14455023
5 - Import and save
6 - Search for the record again
7 - Import and save - duplicate check isn't been triggered
8 - Apply patch
9 - restart_all
10 - repeat 2-7
11 - this time you should get a duplicate notification
12 - Sitch SearchEngine syspref to 'Zebra'
13 - repeat 2-7
14 - Ensure you still get duplicate notification

Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 8 Tomás Cohen Arazi 2023-03-06 12:48:50 UTC
Pushed to master for 23.05.

Nice work everyone, thanks!
Comment 9 Jacob O'Mara 2023-03-07 18:27:43 UTC
Nice work, thanks everyone!

Pushed to 22.11.x for the next release.
Comment 10 Lucas Gass 2023-03-10 16:35:15 UTC
Backported to 22.05.x for upcoming 22.05.11
Comment 11 Arthur Suzuki 2023-03-29 13:13:43 UTC
applied to 21.11 for 21.11.19
Comment 12 wainuiwitikapark 2023-04-27 05:39:22 UTC
Not backported to 21.05.x