Bug 23324

Summary: Need an ISBN normalization routine
Product: Koha Reporter: Nick Clemens <nick>
Component: MARC Bibliographic record staging/importAssignee: Nick Clemens <nick>
Status: CLOSED FIXED QA Contact: Testopia <testopia>
Severity: normal    
Priority: P5 - low CC: fridolin.somers, lucas, m.de.rooy, martin.renvoize
Version: Main   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: Small patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
19.11.00,19.05.04
Bug Depends on: 23322    
Bug Blocks: 23468    
Attachments: Sample import file
Bug 23324: Add an ISBN normalization routine
Bug 23324: Add an ISBN normalization routine
Bug 23324:(QA follow-up) Use existing NormalizeISBN function
Bug 23324: Add an ISBN normalization routine
Bug 23324: (QA follow-up) Use existing NormalizeISBN function
Bug 23324: (QA follow-up) Typo and remove unneeded use statement

Description Nick Clemens 2019-07-16 13:09:27 UTC
This bug was highlighted by use of Elasticsearch.

ES default indexing of ISBN uses Business::ISBN to normalize ISBN in the DB and save all variations. This works great but it does mean that:
9780670026623 (alk. paper)
Is indexed as:
9780670026623

So when importing a record a search for:
9780670026623 (alk. paper)

returns no results.

We need to implement an ISBN normalizer to handle this case
Comment 1 Nick Clemens 2019-07-16 13:10:55 UTC
Created attachment 91550 [details]
Sample import file
Comment 2 Nick Clemens 2019-07-16 13:14:10 UTC
Created attachment 91551 [details] [review]
Bug 23324: Add an ISBN normalization routine

To test:
1 - Set SearchEngine to ElasticSearch
2 - Stage the sample file (import it if it doesn't already exist in your catalog and then stage again)
3 - Set matching rule to ISBN
4 - No matches found
5 - Apply patch
6 - Apply no matchign rule
7 - Change the ISBN matching rule to use ISBN normalizer
8 - Apply matching rule for ISBN
9 - It matches!
Comment 3 Nick Clemens 2019-07-16 16:38:49 UTC
Created attachment 91557 [details] [review]
Bug 23324: Add an ISBN normalization routine

To test:
1 - Set SearchEngine to ElasticSearch
2 - Stage the sample file (import it if it doesn't already exist in your catalog and then stage again)
3 - Set matching rule to ISBN
4 - No matches found
5 - Apply patch
6 - Apply no matchign rule
7 - Change the ISBN matching rule to use ISBN normalizer
8 - Apply matching rule for ISBN
9 - It matches!

Signed-off-by: Ron Houk <rhouk@ottumwapubliclibrary.org>
Comment 4 Marcel de Rooy 2019-08-09 09:47:55 UTC
What about NormalizeISBN in C4/Koha ?
Comment 5 Nick Clemens 2019-08-09 10:49:43 UTC
Created attachment 92116 [details] [review]
Bug 23324:(QA follow-up) Use existing NormalizeISBN function
Comment 6 Nick Clemens 2019-08-09 10:52:01 UTC
(In reply to Marcel de Rooy from comment #4)
> What about NormalizeISBN in C4/Koha ?

I didn't know about that!  I altered these patches to use it. I think eventually the whole routine should be moved here, but I elected to keep this smaller as it affects stable (using ES) and prevents matching imports on ISBN - I added a FIXME
Comment 7 Marcel de Rooy 2019-08-16 06:09:39 UTC
Created attachment 92253 [details] [review]
Bug 23324: Add an ISBN normalization routine

To test:
1 - Set SearchEngine to ElasticSearch
2 - Stage the sample file (import it if it doesn't already exist in your catalog and then stage again)
3 - Set matching rule to ISBN
4 - No matches found
5 - Apply patch
6 - Apply no matchign rule
7 - Change the ISBN matching rule to use ISBN normalizer
8 - Apply matching rule for ISBN
9 - It matches!

Signed-off-by: Ron Houk <rhouk@ottumwapubliclibrary.org>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 8 Marcel de Rooy 2019-08-16 06:09:44 UTC
Created attachment 92254 [details] [review]
Bug 23324: (QA follow-up) Use existing NormalizeISBN function

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 9 Marcel de Rooy 2019-08-16 06:09:48 UTC
Created attachment 92255 [details] [review]
Bug 23324: (QA follow-up) Typo and remove unneeded use statement

Two typos in POD typo for routine name.
Remove "use Business::ISBN" from Normalize.pm.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 10 Martin Renvoize 2019-08-16 11:27:20 UTC
Nice work!

Pushed to master for 19.11.00
Comment 11 Fridolin Somers 2019-09-05 14:56:59 UTC
Pushed to 19.05.x for 19.05.04
Comment 12 Lucas Gass 2019-09-24 15:06:46 UTC
backported to 18.11.x for 18.11.10