Bug 6590 - Removing hyphens from isbn and issn when cataloging a biblio
Summary: Removing hyphens from isbn and issn when cataloging a biblio
Status: In Discussion
Alias: None
Product: Koha
Classification: Unclassified
Component: Cataloging (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Target Milestone: ---
Assignee: Juan Romay Sieira
QA Contact: Bugs List
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-07-15 09:55 UTC by Juan Romay Sieira
Modified: 2016-08-10 15:05 UTC (History)
11 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Bot Control: ---
When did the bot last check this:
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:


Attachments
Patch to remove hyphens (16.55 KB, patch)
2011-07-15 09:59 UTC, Juan Romay Sieira
Details | Diff | Splinter Review
Patch to apply after the first one (5.37 KB, patch)
2011-07-18 10:44 UTC, Juan Romay Sieira
Details | Diff | Splinter Review
Missed add this file misc/bin/zebraqueue_daemon.pl in the second patch (2.53 KB, patch)
2011-07-18 10:54 UTC, Juan Romay Sieira
Details | Diff | Splinter Review
Removing hyphens from isbn and issn (26.38 KB, patch)
2011-07-19 11:35 UTC, Juan Romay Sieira
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Juan Romay Sieira 2011-07-15 09:55:37 UTC
By removing hyphens from isbn and issn in marcxml and marc, it makes possible search for a isbn or issn with or without them.
When a search of an isbn, or issn is made, it has to remove them too.
Comment 1 Juan Romay Sieira 2011-07-15 09:59:53 UTC Comment hidden (obsolete)
Comment 2 Frédéric Demians 2011-07-15 17:43:42 UTC
I don't agree with this patch.

1) It doesn't handle multiple ISBN. In UNIMARC biblio record, you can
   repeat 010 field. With this patch, the first ISBN is 'normalized'.
   The followings are kept as they are.

2) This line doesn't sound good:

   $operand =~ s/-//g if (C4::Context->preference("IsbnIssnRemoveHyphens"));

   '-' doesn't have the same meaning depending of the context. It must
   be interpreted differently in a term and in a ISBN. In an ISBN, it
   must be suppressed to concatenate all ISBN parts. In a phrase, it is
   a breaking character like space, comma, etc. 

3) With this patch, ISBN stored in Koha DB don't follow anymore
   MARC21/UNIMARC format which specify that hyphens must be present.

In my opinion, correct ISBN indexing and search should be delegated to
Zebra and mustn't modify stored biblio records. We could create a new
index type (we already have word, phrase, numeric index types). For this
type, hyphens could be suppressed by Zebra. And Koha search query
parser could identify ISBN/ISSN by a regex and suppress hyphens only for
them before sending search query to Zebra.
Comment 3 Magnus Enger 2011-07-15 17:45:58 UTC
Thanks for remembering NORMARC! ;-) 

ISBNs and ISSNs are treated the same way in NORMARC and MARC21, so this: 

   } elsif ($encoding eq 'MARC21') {
   ...
   } elsif ($encoding eq 'NORMARC') {

could just as well be turned into this: 

   } elsif ($encoding eq 'MARC21' || $encoding eq 'NORMARC') {

to avoid some redundant code. Other places in the code treat MARC21 as the
default, so this could be simplified even further, to this: 

   } else {

Just a suggestion!
Comment 4 Juan Romay Sieira 2011-07-15 21:25:01 UTC
(In reply to comment #2)
> I don't agree with this patch.
> 
> 1) It doesn't handle multiple ISBN. In UNIMARC biblio record, you can
>    repeat 010 field. With this patch, the first ISBN is 'normalized'.
>    The followings are kept as they are.
> 
> 2) This line doesn't sound good:
> 
>    $operand =~ s/-//g if (C4::Context->preference("IsbnIssnRemoveHyphens"));
> 
>    '-' doesn't have the same meaning depending of the context. It must
>    be interpreted differently in a term and in a ISBN. In an ISBN, it
>    must be suppressed to concatenate all ISBN parts. In a phrase, it is
>    a breaking character like space, comma, etc. 
> 
> 3) With this patch, ISBN stored in Koha DB don't follow anymore
>    MARC21/UNIMARC format which specify that hyphens must be present.
> 
> In my opinion, correct ISBN indexing and search should be delegated to
> Zebra and mustn't modify stored biblio records. We could create a new
> index type (we already have word, phrase, numeric index types). For this
> type, hyphens could be suppressed by Zebra. And Koha search query
> parser could identify ISBN/ISSN by a regex and suppress hyphens only for
> them before sending search query to Zebra.

If UNIMARC handle multiple isbns, its no just a problem to loop over them, and remove hyphens in every one. And, the original isbn is kept in isbn field in biblioitems table.
When a search is performed, only during search specifically by isbn or issn, hyphens are removed from query, but not during other search like kw, au ...
I'm agree with the modification in koha. Add another index type to handle this. I saw somebody did it a few days ago, here: http://lists.indexdata.dk/pipermail/zebralist/2010-May/002423.html , it maybe work, its only a decision about handle it.

We need this change, by one way or another, because most of the people search an isbn without hyphens, and as I saw, most of the Z we used, could handle this, searching with or without hyphens.
Comment 5 Frédéric Demians 2011-07-16 08:45:02 UTC
> If UNIMARC handle multiple isbns, its no just a problem to loop over
> them, and remove hyphens in every one. And, the original isbn is kept
> in isbn field in biblioitems table.

But (1) when a library will export its catalog in ISO2709, ISBN fields
will come from biblioitems.marcxml not biblioitems.isbn and (2)
biblioitems.isbn size is only 30 characters which could be too short
with multiple ISBNs.

> When a search is performed, only during search specifically by isbn or
> issn, hyphens are removed from query, but not during other search like
> kw, au ...

Ok, so it's not that bad but it would be great if a anywhere search
would work also with ISBN.

> I'm agree with the modification in koha. Add another index type to
> handle this.  I saw somebody did it a few days ago, here:

> http://lists.indexdata.dk/pipermail/zebralist/2010-May/002423.html , it maybe
> work, its only a decision about handle it.

We can ask to Frydolin. I've added him to this thread.

> We need this change, by one way or another, because most of the people
> search an isbn without hyphens, and as I saw, most of the Z we used,
> could handle this, searching with or without hyphens.

I agree and I support 100% your work. I really want it into Koha but we
also need to avoid a regression. I know libraries who count on Koha to
catalog normalized UNIMARC biblio records.
Comment 6 Juan Romay Sieira 2011-07-18 10:44:20 UTC Comment hidden (obsolete)
Comment 7 Juan Romay Sieira 2011-07-18 10:54:49 UTC Comment hidden (obsolete)
Comment 8 Fridolin SOMERS 2011-07-18 14:59:19 UTC
Hie,

This is the modifications whe made for Lyon2 University :

http://git.progilone.com/?p=koha-lyon2.git;a=commitdiff;h=f4866b2c3ceb57403c4cf3359ae3c2ded48a6912

It works, but I think it is not a real solution.
And it is not compatible with 'nosanitize' mode, witch is used to fasten indexation.

In my opinion, ISBN should has a constant format.
Can't it be managed with Business::ISBN ?

http://search.cpan.org/dist/Business-ISBN/
Comment 9 Juan Romay Sieira 2011-07-19 11:35:43 UTC
Created attachment 4667 [details] [review]
Removing hyphens from isbn and issn

If syspref IsbnIssnRemoveHyphens is enabled, hyphens from isbn and issn are removed when zebra operations are executed. With rebuild_zebra does the same thing, and it could be executed with -x -nosanitize param or not.

To test it, apply this patch, insert the new syspref & enable it, replace zebraqueue_daemon.pl (of xxx-dev folder) with the included in misc/bin/, and restart koha-zebraqueue-ctl.sh. Catalog a biblio or rebuild zebra index.
Comment 10 Julian Maurice 2011-07-22 08:47:10 UTC
Patch seems to work on my Unimarc install.
I tried to modify an existing record and add hyphens in isbn and issn, and then search for these isbn and issn (with and without hyphens).
I tried rebuild_zebra too (with and without -x and -nosanitize options).
Comment 11 Paul Poulain 2011-10-06 12:06:33 UTC
sorry for being late to QA this patch

* the syspref system has been updated recently, and everything is not in installer/data/mysql/sysprefs.sql, no need to have a specific line for each language, so this patch won't apply

* i'm not sure a syspref is needed : when should we keep isbns ? Shouldn't we just remove them.

Otherwise, the patch is good, and if you provide a new patch or convince me that the syspref is needed before 3.6 i'll mark passed QA quickly.
Comment 12 Ian Walls 2011-10-06 12:22:33 UTC
Alternate implementation idea:  use Zebra indexing configs to index both the plaintext ISBN (as it's written in the MARC record) and the normalized ISBN.  This way, we don't have to change any of the actual data in the records or how search works; we just add more access points to the data.  Less chance of regression this way.

I think if we added isbn.chr file to etc/zebradb/etc, updated default.idx to include it, and modified record.abs to add ISBN and ISSN to those index types, we could pull this off.
Comment 13 Juan Romay Sieira 2011-10-06 14:38:19 UTC
The syspref it's only to give an option to end user. If a library wants to keep hyphens indexed, they could.
If you consider to remove it, I can do it, and upload a new patch.
Comment 14 Paul Poulain 2011-10-07 08:22:01 UTC
Ian, Frydolin, anyone : do you see a case where it could be interesting to keep - in isbn/issn ?
Comment 15 Frédéric Demians 2011-10-07 08:51:43 UTC
> Ian, Frydolin, anyone : do you see a case where it could be
> interesting to keep - in isbn/issn ?

In UNIMARC, ISBN field is supposed to contain ISBN as written on the
book cover. ISBN without hyphens can be stored in EAN (073) field.
Comment 16 Fridolin SOMERS 2011-10-07 12:22:18 UTC
(In reply to comment #14)
> Ian, Frydolin, anyone : do you see a case where it could be interesting to keep
> - in isbn/issn ?

There may be a problem :
Hyphens are removed only if ISBN or ISSN index is present in search query.
If no index is selected and ISBN value (with hyphens) entered in search field, search will not find.

This may appear in a copy/paste from an external source.

In our development, we indexed both with and without hyphens by duplicating the ISBN/ISSN sub-fields in rebuild_zebra.

Note : in this patch, ISBN/ISSN field/subfield are hardcoded, regardless to "koha to marc mapping".

(in this case, a more extendable search engine like solr would be welcome)
Comment 17 Ian Walls 2011-10-07 15:49:15 UTC
Since this patch changes the MARC data, and that's in violation of UNIMARC rules, I'm marking this as Failed QA.

I think my solution of adding an additional index field to ISBN that normalizes it would be the most robust; it would work with keyword index, and not alter the data in any way.  It would just provide more access points.  The new .chr file could be reused on any numeric field we need to normalize.
Comment 18 Paul Poulain 2011-10-07 16:05:28 UTC
Ian, agreed with your arguments & conclusion
(& also agreeing that something easier to extend like Solr will be good, but that's another topic ;-) )
Comment 19 Magnus Enger 2011-10-07 16:26:59 UTC
I agree with Ian - Koha should not change stuff that cataloguers put into the MARC records.
Comment 20 Magnus Enger 2013-06-10 19:55:12 UTC
(In reply to comment #12)
> Alternate implementation idea:  use Zebra indexing configs to index both the
> plaintext ISBN (as it's written in the MARC record) and the normalized ISBN.
> This way, we don't have to change any of the actual data in the records or
> how search works; we just add more access points to the data.  Less chance
> of regression this way.
> 
> I think if we added isbn.chr file to etc/zebradb/etc, updated default.idx to
> include it, and modified record.abs to add ISBN and ISSN to those index
> types, we could pull this off.

This sounds promising, but we need to be careful how we do the normalization. We could extract {0-9xX} from 020$a, but then this would result in an incorrect ISBN being indexed:

020 	$a0914378260 (pbk. : v. 1) :$c$5.00

(This example is taken from http://lists.indexdata.dk/pipermail/zebralist/2009-March/002136.html)

So we need to do something like extract the first "word" from 020$a, and then extract {0-9xX} from it. Can we do that with a .chr file? 

Maybe it is easier with DOM indexing, as hinted at here? http://lists.indexdata.dk/pipermail/zebralist/2009-March/002137.html

Or could we do some trickery before the indexing? Like add a Koha-specific, hidden subfield to 020 where we store a normalized ISBN when a record is saved/updated? Business::ISBN would be good for doing that, I think.
Comment 21 Fridolin SOMERS 2013-06-11 08:34:14 UTC
In Biblibre Solr version of Koha, there are plugins that can be defined on (sub)fields to perform such operations before indexing. One exists for ISBN adding normalized and without hyphens forms.

Note that when using ICU indexing, the hyphen is a special separator so search with or without hyphens matchs ISBN, even adding hyphens in random place.
Comment 22 Kyle M Hall 2013-06-20 13:24:21 UTC
> Or could we do some trickery before the indexing? Like add a Koha-specific,
> hidden subfield to 020 where we store a normalized ISBN when a record is
> saved/updated? Business::ISBN would be good for doing that, I think.

This. Business::ISBN does an excellent job of normalizing ISBNs.
Comment 23 David Cook 2015-02-03 01:11:01 UTC
(In reply to Fridolin SOMERS from comment #21)
> In Biblibre Solr version of Koha, there are plugins that can be defined on
> (sub)fields to perform such operations before indexing. One exists for ISBN
> adding normalized and without hyphens forms.
> 
> Note that when using ICU indexing, the hyphen is a special separator so
> search with or without hyphens matchs ISBN, even adding hyphens in random
> place.

I don't really understand ICU indexing, but it looks like there is a specific transliteration rule in the master words-icu.xml for hyphens when they're preceded by numbers:

http://userguide.icu-project.org/transforms/general#TOC-Context

http://git.koha-community.org/gitweb/?p=koha.git;a=blob;f=etc/zebradb/etc/words-icu.xml;h=57498cbca4a3ea2c76399f424b792e3055dcea01;hb=HEAD

It looks to me that hyphens get removed...

But then as Colin Campbell points out, there is a problem searching using hyphens when using ICU:

http://lists.indexdata.dk/pipermail/zebralist/2013-August/002576.html
Comment 24 David Cook 2015-02-03 06:30:03 UTC
I've also run into the same problem as Colin... although I've discovered that it appears to be an issue with newer versions of Zebra such as 2.0.59.

Older versions such as 2.0.47 and 2.0.44 (this latter is the default in the Debian and Ubuntu repositories) seem to be unaffected. So if you installed Zebra via the Indexdata repositories or newer versions by source, you  might be affected by this bug as well...