Bug 23324 - Need an ISBN normalization routine
Summary: Need an ISBN normalization routine
Status: Pushed to oldstable
Alias: None
Product: Koha
Classification: Unclassified
Component: MARC Bibliographic record staging/import (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: Nick Clemens
QA Contact: Testopia
URL:
Keywords:
Depends on: 23322
Blocks: 23468
  Show dependency treegraph
 
Reported: 2019-07-16 13:09 UTC by Nick Clemens
Modified: 2019-09-24 17:52 UTC (History)
4 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:
19.11.00,19.05.04


Attachments
Sample import file (1.47 KB, application/x-wine-extension-mrc)
2019-07-16 13:10 UTC, Nick Clemens
Details
Bug 23324: Add an ISBN normalization routine (5.80 KB, patch)
2019-07-16 13:14 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 23324: Add an ISBN normalization routine (5.86 KB, patch)
2019-07-16 16:38 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 23324:(QA follow-up) Use existing NormalizeISBN function (3.10 KB, patch)
2019-08-09 10:49 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 23324: Add an ISBN normalization routine (5.95 KB, patch)
2019-08-16 06:09 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 23324: (QA follow-up) Use existing NormalizeISBN function (3.22 KB, patch)
2019-08-16 06:09 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 23324: (QA follow-up) Typo and remove unneeded use statement (1.23 KB, patch)
2019-08-16 06:09 UTC, Marcel de Rooy
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Nick Clemens 2019-07-16 13:09:27 UTC
This bug was highlighted by use of Elasticsearch.

ES default indexing of ISBN uses Business::ISBN to normalize ISBN in the DB and save all variations. This works great but it does mean that:
9780670026623 (alk. paper)
Is indexed as:
9780670026623

So when importing a record a search for:
9780670026623 (alk. paper)

returns no results.

We need to implement an ISBN normalizer to handle this case
Comment 1 Nick Clemens 2019-07-16 13:10:55 UTC
Created attachment 91550 [details]
Sample import file
Comment 2 Nick Clemens 2019-07-16 13:14:10 UTC
Created attachment 91551 [details] [review]
Bug 23324: Add an ISBN normalization routine

To test:
1 - Set SearchEngine to ElasticSearch
2 - Stage the sample file (import it if it doesn't already exist in your catalog and then stage again)
3 - Set matching rule to ISBN
4 - No matches found
5 - Apply patch
6 - Apply no matchign rule
7 - Change the ISBN matching rule to use ISBN normalizer
8 - Apply matching rule for ISBN
9 - It matches!
Comment 3 Nick Clemens 2019-07-16 16:38:49 UTC
Created attachment 91557 [details] [review]
Bug 23324: Add an ISBN normalization routine

To test:
1 - Set SearchEngine to ElasticSearch
2 - Stage the sample file (import it if it doesn't already exist in your catalog and then stage again)
3 - Set matching rule to ISBN
4 - No matches found
5 - Apply patch
6 - Apply no matchign rule
7 - Change the ISBN matching rule to use ISBN normalizer
8 - Apply matching rule for ISBN
9 - It matches!

Signed-off-by: Ron Houk <rhouk@ottumwapubliclibrary.org>
Comment 4 Marcel de Rooy 2019-08-09 09:47:55 UTC
What about NormalizeISBN in C4/Koha ?
Comment 5 Nick Clemens 2019-08-09 10:49:43 UTC
Created attachment 92116 [details] [review]
Bug 23324:(QA follow-up) Use existing NormalizeISBN function
Comment 6 Nick Clemens 2019-08-09 10:52:01 UTC
(In reply to Marcel de Rooy from comment #4)
> What about NormalizeISBN in C4/Koha ?

I didn't know about that!  I altered these patches to use it. I think eventually the whole routine should be moved here, but I elected to keep this smaller as it affects stable (using ES) and prevents matching imports on ISBN - I added a FIXME
Comment 7 Marcel de Rooy 2019-08-16 06:09:39 UTC
Created attachment 92253 [details] [review]
Bug 23324: Add an ISBN normalization routine

To test:
1 - Set SearchEngine to ElasticSearch
2 - Stage the sample file (import it if it doesn't already exist in your catalog and then stage again)
3 - Set matching rule to ISBN
4 - No matches found
5 - Apply patch
6 - Apply no matchign rule
7 - Change the ISBN matching rule to use ISBN normalizer
8 - Apply matching rule for ISBN
9 - It matches!

Signed-off-by: Ron Houk <rhouk@ottumwapubliclibrary.org>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 8 Marcel de Rooy 2019-08-16 06:09:44 UTC
Created attachment 92254 [details] [review]
Bug 23324: (QA follow-up) Use existing NormalizeISBN function

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 9 Marcel de Rooy 2019-08-16 06:09:48 UTC
Created attachment 92255 [details] [review]
Bug 23324: (QA follow-up) Typo and remove unneeded use statement

Two typos in POD typo for routine name.
Remove "use Business::ISBN" from Normalize.pm.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 10 Martin Renvoize 2019-08-16 11:27:20 UTC
Nice work!

Pushed to master for 19.11.00
Comment 11 Fridolin SOMERS 2019-09-05 14:56:59 UTC
Pushed to 19.05.x for 19.05.04
Comment 12 Lucas Gass 2019-09-24 15:06:46 UTC
backported to 18.11.x for 18.11.10