Bug 15187 - Adding 880 Fields to index-list in order to Increase Search for ALL non-latin Scripts
Summary: Adding 880 Fields to index-list in order to Increase Search for ALL non-latin...
Status: RESOLVED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: master
Hardware: All All
: P1 - high normal (vote)
Assignee: David Cook
QA Contact: Martin Renvoize
URL:
Keywords: rel_22_05_candidate
: 17407 (view as bug list)
Depends on: 31532
Blocks: 32033
  Show dependency treegraph
 
Reported: 2015-11-13 15:23 UTC by Jesse Lambertson
Modified: 2023-12-28 20:44 UTC (History)
15 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Medium patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
This fixes the Zebra search engine when using ICU* so that 880 fields are rewritten as their linked fields and the alternate graphic representation of fields are indexed, in the same way that it works for Elasticsearch. Example: add 245-01 to 880$6 and 教牧書信 to 880$a - the Chinese characters are now indexed into the title index using the 245 rules. * ICU is a feature of the Zebra search engine that can be configured to make searching with non-latin languages (such as Chinese and Arabic) work correctly.
Version(s) released in:
22.11.00, 22.05.05


Attachments
Zipped file of MARC records (150 bytes, application/x-zip-compressed)
2015-11-13 16:35 UTC, Jesse Lambertson
Details
MARC upload with a few examples with 880s (13.58 KB, application/octet)
2015-11-16 15:11 UTC, Jesse Lambertson
Details
Bug 15187: Index 880 in Zebra the same as Elasticsearch (4.53 KB, patch)
2022-05-02 07:40 UTC, David Cook
Details | Diff | Splinter Review
Bug 15187: Index 880 in Zebra the same as Elasticsearch (4.57 KB, patch)
2022-07-03 21:28 UTC, David Nind
Details | Diff | Splinter Review
Bug 15187: Index 880 in Zebra the same as Elasticsearch (4.64 KB, patch)
2022-07-06 11:54 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 15187: (QA follow-up) Add unit test (961 bytes, patch)
2022-07-06 11:54 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 15187: Index 880 in Zebra the same as Elasticsearch (4.64 KB, patch)
2022-07-06 17:08 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 15187: (QA follow-up) Add unit test (3.48 KB, patch)
2022-07-06 17:08 UTC, Martin Renvoize
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Jesse Lambertson 2015-11-13 15:23:57 UTC
We have found that after we turned on ICU indexing to index Arabic in our catalogue (using these steps: http://wiki.koha-community.org/wiki/Correcting_Search_of_Arabic_records), we get results from searches in Arabic but NOT in specific search-limits such as "title," "author" or "series."

This is 880 isn't INDEXED SPECIFICALLY TO DELIVER SEARCH RESULTS.

We would very much like to add this functionality to our Koha instance.

We have tried our hand at these steps (to add new field to the indexing list): http://wiki.koha-community.org/wiki/How_to_add_new_zebra_index
to no avail. 

A friendly listserver suggested something like this:

> <xsl:template match="marc:datafield[@tag='880']">
>       <xsl:variable name="linkage"
> select="substring(marc:subfield[@code='6']",1,3)"/><!-- linkage might
> contain something like 245 or 260 or 100 -->
>       <xsl:call-template name="$linkage" />
> </xsl:template>

But then suggested that we try something like this:

>   <index_subfields tag="880" subfields="a"
> condition="substring(marc:subfield[@code='6']",1,3) = 245">
>     <target_index>Title-cover:w</target_index>
>     <target_index>Title-cover:p</target_index>
>     <target_index>Title-cover:s</target_index>
>     <target_index>Title:w</target_index>
>     <target_index>Title:p</target_index>
>     <target_index>Title:s</target_index>
>   </index_subfields>

But that this required a specific block of edits to be made FOR EVERY LINKED FIELD to get 880 indexed "properly."

Thank you.

I have attached a few records from or catalogue.

Jesse Lambertson
jlambertson@sqcc.org
Comment 1 Jesse Lambertson 2015-11-13 16:35:14 UTC Comment hidden (obsolete)
Comment 2 Bernardo Gonzalez Kriegel 2015-11-13 17:27:34 UTC
Hi Jesse,
may be I don't know which is the best solution to your problem, but as far as I understand 880 is not only to titles, but for any field that needs another graphical representation.

I did a test with and Arabic title in a 246 field as parallel title, and I can found it from staff/opac as title without modifications.

BTW, your zip file have a zero byte record.
Comment 3 Jesse Lambertson 2015-11-13 17:50:34 UTC
(In reply to Bernardo Gonzalez Kriegel from comment #2)
> Hi Jesse,
> may be I don't know which is the best solution to your problem, but as far
> as I understand 880 is not only to titles, but for any field that needs
> another graphical representation.
> 
> I did a test with and Arabic title in a 246 field as parallel title, and I
> can found it from staff/opac as title without modifications.
> 
> BTW, your zip file have a zero byte record.

I will reupload the MARC file then just for examples from our records. 
Thank you.

Strange,

Regarding the Arabic searching.

You are correct that the issue affects all Arabic script in 880 fields.
245 and 246 will search because they are ALREADY indexed.

880s are not indexed separately from "keyword" searching so we can't search Arabic in any other way than in "keyword."

Thank you,

Jesse
Comment 4 Jesse Lambertson 2015-11-16 15:11:51 UTC
Created attachment 44876 [details]
MARC upload with a few examples with 880s

Faulty upload before, this file has a few records in it.
Comment 5 Katrin Fischer 2016-02-02 07:10:32 UTC
While this is needed for libraries with original script that use 880 for cataloguing, I think it's not a critical bug. Koha hasn't supported this so far, although it would be nice if it would. With DOM and the recently added options for conditional indexing it should be finally possible to make this work nicely.
Comment 6 Jon Knight 2018-08-23 09:36:54 UTC
If this is not viewed as an important bug, can we drop the Importance "P1 - high" to be a lower category?
Comment 7 Katrin Fischer 2018-09-21 06:02:39 UTC
(In reply to Jon Knight from comment #6)
> If this is not viewed as an important bug, can we drop the Importance "P1 -
> high" to be a lower category?

Hi Jon, we are not using the Importance field at the moment. It would be ok to just change it.
Comment 8 Katrin Fischer 2019-06-10 08:38:34 UTC
*** Bug 17407 has been marked as a duplicate of this bug. ***
Comment 9 David Cook 2022-05-01 23:58:26 UTC
I'll be looking more at this soon. The solution I have in mind won't be as elegant as I would like, but it should be reasonably functional...
Comment 10 David Cook 2022-05-02 02:52:13 UTC
Oh... actually this looks like it might be very difficult to do within the community model without making substantial changes to ./etc/zebradb/xsl/koha-indexdefs-to-zebra.xsl and ./etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml. These files were designed with too simple a logic in mind.

I'm sorry but I don't have the resources to make those systematic architectural changes.

I'm planning instead to locally customize ./etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl to have customized "match" attributes.

For example:

Replacing the following:
<xslo:template mode="index_data_field" match="marc:datafield[@tag='245']">

With the following:
<xslo:template mode="index_data_field" match="marc:datafield[@tag='245' or (@tag='880' and (marc:subfield[@code=6 and substring(text(),1,3)='245']))]">

While at a glance it's a very simple change, it's impossible to make that change using ./etc/zebradb/xsl/koha-indexdefs-to-zebra.xsl and ./etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml to create ./etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl
Comment 11 David Cook 2022-05-02 03:05:09 UTC
I suppose you could customize ./etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml so that the 880 fields mirror the potential linkages (like 245), but it would be very easy for them to fall out of sync. 

My method of using a more complex "match" attribute is easier to maintain, although it does still run into the problem of needing to add the 880 logic for each possible mapping, which is labour-intensive and error-prone.

Jesse, your original idea has merit, but it won't work with how the Koha Zebra indexing XSLT has been designed. 

Although I wonder if I can think of something else clever...
Comment 12 David Cook 2022-05-02 04:02:15 UTC
(In reply to David Cook from comment #11)
> Although I wonder if I can think of something else clever...

Another thought is to do more transformations in the Zebra DOM pipeline... either by adding an "input" pipeline or by adding more "Extract" transform documents. (See https://software.indexdata.com/zebra/doc/idzebra.pdf) 

The idea would be that the 880 $6 245-01 would be copied into a 245 and that way it could just be fed through the normal indexing process. 

Another thing to think about is how this is Zebra-specific. I don't know what would need to be done for Elasticsearch (if anything)...
Comment 13 David Cook 2022-05-02 04:47:35 UTC
It's a bit of a hack and I haven't tried it yet with the Zebra pipelines yet, but I'll include my transform here. It copies all the existing XML elements, but for 880 elements it creates a new element based off the linkage, and it just copies the indicators and subfields into that new element according to the MARC bib spec.

I would've liked to have integrated this into Koha's existing XSLTs but I couldn't see an easy way of doing so, although I have a few ideas about how maybe it could work... 

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:marc="http://www.loc.gov/MARC21/slim"
 version="1.0">
<xsl:output indent="yes"
      method="xml"
      version="1.0"
      encoding="UTF-8"/>

 <xsl:template name="identity" match="node()|@*">
   <xsl:copy>
     <xsl:apply-templates select="@*|node()"/>
   </xsl:copy>
 </xsl:template>

 <xsl:template match="marc:datafield[@tag=880]">
    <xsl:call-template name="identity"/>
    <xsl:variable name="tag" select="substring(marc:subfield[@code=6],1,3)"/>
    <xsl:text>&#xa;</xsl:text>
    <xsl:text>  </xsl:text>
    <xsl:element name="datafield" namespace="http://www.loc.gov/MARC21/slim">
        <xsl:attribute name="tag"><xsl:value-of select="substring(marc:subfield[@code=6],1,3)"/></xsl:attribute>
        <xsl:attribute name="ind1"><xsl:value-of select="@ind1"/></xsl:attribute>
        <xsl:attribute name="ind2"><xsl:value-of select="@ind2"/></xsl:attribute>
        <xsl:apply-templates select="marc:subfield[@code != '6']"/>
    </xsl:element>
</xsl:template>

</xsl:stylesheet>
Comment 14 David Cook 2022-05-02 05:54:49 UTC
(In reply to David Cook from comment #13)
> <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
> xmlns:marc="http://www.loc.gov/MARC21/slim"
>  version="1.0">
> <xsl:output indent="yes"
>       method="xml"
>       version="1.0"
>       encoding="UTF-8"/>
> 
>  <xsl:template name="identity" match="node()|@*">
>    <xsl:copy>
>      <xsl:apply-templates select="@*|node()"/>
>    </xsl:copy>
>  </xsl:template>
> 
>  <xsl:template match="marc:datafield[@tag=880]">
>     <xsl:call-template name="identity"/>
>     <xsl:variable name="tag" select="substring(marc:subfield[@code=6],1,3)"/>
>     <xsl:text>&#xa;</xsl:text>
>     <xsl:text>  </xsl:text>
>     <xsl:element name="datafield" namespace="http://www.loc.gov/MARC21/slim">
>         <xsl:attribute name="tag"><xsl:value-of
> select="substring(marc:subfield[@code=6],1,3)"/></xsl:attribute>
>         <xsl:attribute name="ind1"><xsl:value-of
> select="@ind1"/></xsl:attribute>
>         <xsl:attribute name="ind2"><xsl:value-of
> select="@ind2"/></xsl:attribute>
>         <xsl:apply-templates select="marc:subfield[@code != '6']"/>
>     </xsl:element>
> </xsl:template>
> 
> </xsl:stylesheet>

Steps to get this to work in a instance-specific context:

1. I saved the above as /etc/koha/zebradb/xsl/preprocess_marcxml.xsl
2. cp /etc/koha/zebradb/biblios/etc/dom-config.xml /etc/koha/sites/kohadev/test-dom-config.xml
3. vi /etc/koha/sites/kohadev/zebra-biblios-dom.cfg
#For marcxml, I replaced /etc/koha/zebradb/biblios/etc/dom-config.xml with /etc/koha/sites/kohadev/test-dom-config.xml
4. vi /etc/koha/sites/kohadev/test-dom-config.xml
#Use the following:
<dom>
  <extract name="index">
    <xslt stylesheet="preprocess_marcxml.xsl"/>
    <xslt stylesheet="biblio-zebra-indexdefs.xsl"/>
  </extract>
  <retrieve name="marc">
    <xslt stylesheet="identity.xsl" />
  </retrieve>
  <retrieve name="marcxml">
    <xslt stylesheet="identity.xsl" />
  </retrieve>
  <retrieve name="zebra">
    <xslt stylesheet="zebra.xsl"/>
  </retrieve>
  <retrieve name="index">
    <xslt stylesheet="biblio-zebra-indexdefs.xsl"/>
  </retrieve>
  <input>
    <xmlreader level="1"/>
  </input>
</dom>

5. koha-rebuild-zebra -f -b -v kohadev
Comment 15 David Cook 2022-05-02 05:58:39 UTC
The results are amazing! It works exactly like I hoped!

I'm looking at a English/Chinese record that has many 880s for a variety of different fields. While Koha's default XSLTs will surface the 880 data on the detail page, *without this change*, the links don't work. 

However, with this change, they all work as you'd expect! Whether you click on the English or the Chinese, it's very smooth. 

--

Note that I haven't benchmarked this in a significant way. However, indexing 400 records in koha-testing-docker... there's no observable difference.
Comment 16 David Cook 2022-05-02 06:41:24 UTC
By the way, it looks like our indexing code for Elasticsearch already handles 880s:

my $tag = $field->tag();
# Handle alternate scripts in MARC 21
my $altscript = 0;
if ($marcflavour eq 'marc21' && $tag eq '880') {
    my $sub6 = $field->subfield('6');
    if ($sub6 =~ /^(...)-\d+/) {
        $tag = $1;
        $altscript = 1;
    }
}

In that case, it looks like they're not indexing 880 fields at all. Perhaps that makes sense since if you index the 880 and the linkage derived from it, you'd end up with 2x the keyword indexed fields...
Comment 17 David Cook 2022-05-02 07:00:57 UTC
Of course, this transform is MARC21 specific, so it needs to be done in a way that is compatible with UNIMARC...
Comment 18 David Cook 2022-05-02 07:14:53 UTC
(In reply to David Cook from comment #17)
> Of course, this transform is MARC21 specific, so it needs to be done in a
> way that is compatible with UNIMARC...

I suppose another way to do this would be to change ./misc/migration_tools/rebuild_zebra.pl to use the same logic as ./Koha/SearchEngine/Elasticsearch.pm...
Comment 19 David Cook 2022-05-02 07:40:37 UTC
Created attachment 134457 [details] [review]
Bug 15187: Index 880 in Zebra the same as Elasticsearch

This patch modifies the MARC21 export to Zebra, so that 880 fields
are rewritten as their linked fields, in the same way that we
already do with Elasticsearch, so that the alternate graphic
representation of fields are indexed accordingly. (ie 880 $6245-01
Chinese titles will be indexed into the title index using the 245 rules)

Test plan:
0. Apply patch
1. Turn on ICU indexing
1b. vi /etc/koha/zebradb/etc/default.idx
1c. Replace charmap word-phrase-utf.chr with icuchain words-icu.xml
1d. Replace charmap word-phrase-utf.chr with icuchain phrases-icu.xml
1e. Restart Zebra server
1f. Re-index Zebra
2. Add record with a 880 $6 245-01 $a 教牧書信 field.
3. Search for this record using a title index with the Chinese title
4. Note that the record is correctly retrieved

(Note: This test probably works better using author or series as they
 present as links on the detail page which makes the fix more obviously
 useful.)
Comment 20 David Nind 2022-07-03 21:28:26 UTC
Created attachment 137050 [details] [review]
Bug 15187: Index 880 in Zebra the same as Elasticsearch

This patch modifies the MARC21 export to Zebra, so that 880 fields
are rewritten as their linked fields, in the same way that we
already do with Elasticsearch, so that the alternate graphic
representation of fields are indexed accordingly. (ie 880 $6245-01
Chinese titles will be indexed into the title index using the 245 rules)

Test plan:
0. Apply patch
1. Turn on ICU indexing
1b. vi /etc/koha/zebradb/etc/default.idx
1c. Replace charmap word-phrase-utf.chr with icuchain words-icu.xml
1d. Replace charmap word-phrase-utf.chr with icuchain phrases-icu.xml
1e. Restart Zebra server
1f. Re-index Zebra
2. Add record with a 880 $6 245-01 $a 教牧書信 field.
3. Search for this record using a title index with the Chinese title
4. Note that the record is correctly retrieved

(Note: This test probably works better using author or series as they
 present as links on the detail page which makes the fix more obviously
 useful.)

Signed-off-by: David Nind <david@davidnind.com>
Comment 21 David Nind 2022-07-03 22:02:46 UTC
Testing notes (using koha-testing-docker):

1. Setting up ICU: see comment 16 (notes 1 and 2) for bug 11158

2. Needed to update the bibliographic framework for books (BKS) to make 880$6 and 880$a visible in the editor.

3. Not related to this bug, but I did try to import the attached MARC records, but it stalled on Staged MARC record management screen, and still shows as importing (there are some other current bugs about issues with importing records).

Error in the logs:

[2022/07/03 21:25:48] [WARN] Filehandle STDOUT reopened as $fh only for input at /usr/share/perl5/DateTime/TimeZone/Local/Unix.    pm line 155.
  2  main::__ANON__ /kohadevbox/koha/tools/manage-marc-import.pl (331)
  3 [2022/07/03 21:25:48] [WARN] DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::mysql::st execute failed: Cannot ad    d or update a child row: a foreign key constraint fails (`koha_kohadev`.`items`, CONSTRAINT `items_ibfk_2` FOREIGN KEY (`homebr    anch`) REFERENCES `branches` (`branchcode`) ON UPDATE CASCADE) at /kohadevbox/koha/Koha/Object.pm line 170
  4  main::__ANON__ /kohadevbox/koha/tools/manage-marc-import.pl (331)

4. I had a go at the release notes text - feel free to edit!
Comment 22 Martin Renvoize 2022-07-06 11:22:54 UTC
This is really nice and elegant David!  QAing now.
Comment 23 Martin Renvoize 2022-07-06 11:54:44 UTC Comment hidden (obsolete)
Comment 24 Martin Renvoize 2022-07-06 11:54:50 UTC Comment hidden (obsolete)
Comment 25 Martin Renvoize 2022-07-06 11:57:38 UTC
I added a unit test for you to get this through QA.. Tomas may want to rename the filter at push.. I couldn't come up with a nicer name.. but I don't really like they we call out it's a Zebra thing... 

What I've not managed to test.. does this cause any issues with non-icu indexed Koha's.. I imagine not as you are unlikely to have such a field in your database in the first place without having enabled ICU (all our customers are ICU by default in case).

Passing QA.. scripts are happy, we're now covered by tests and it's a great improvement.
Comment 26 Martin Renvoize 2022-07-06 17:08:17 UTC
Created attachment 137233 [details] [review]
Bug 15187: Index 880 in Zebra the same as Elasticsearch

This patch modifies the MARC21 export to Zebra, so that 880 fields
are rewritten as their linked fields, in the same way that we
already do with Elasticsearch, so that the alternate graphic
representation of fields are indexed accordingly. (ie 880 $6245-01
Chinese titles will be indexed into the title index using the 245 rules)

Test plan:
0. Apply patch
1. Turn on ICU indexing
1b. vi /etc/koha/zebradb/etc/default.idx
1c. Replace charmap word-phrase-utf.chr with icuchain words-icu.xml
1d. Replace charmap word-phrase-utf.chr with icuchain phrases-icu.xml
1e. Restart Zebra server
1f. Re-index Zebra
2. Add record with a 880 $6 245-01 $a 教牧書信 field.
3. Search for this record using a title index with the Chinese title
4. Note that the record is correctly retrieved

(Note: This test probably works better using author or series as they
 present as links on the detail page which makes the fix more obviously
 useful.)

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 27 Martin Renvoize 2022-07-06 17:08:21 UTC
Created attachment 137234 [details] [review]
Bug 15187: (QA follow-up) Add unit test

This patch adds a unit test for the newly introduced filter.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 28 Tomás Cohen Arazi 2022-07-22 18:04:58 UTC
I hate the filter name. But it is accurate!
Comment 29 Tomás Cohen Arazi 2022-07-22 18:18:58 UTC
Pushed to master for 22.11.

Nice work everyone, thanks!
Comment 30 Lucas Gass 2022-09-02 18:02:19 UTC
Backported to 22.05.x for 22.05.05
Comment 31 David Cook 2022-09-08 04:32:24 UTC
Ugh, this actually causes a regression with the processing of the 880 in Zebra search results, because we're fetching the MARCXML from Zebra instead of MySQL.

I think we'll need to revert this change, and explore my Zebra-based one instead I think.
Comment 32 Arthur Suzuki 2022-09-14 09:50:29 UTC
Hi David,
Given your last comment I won't backport to 21.11 until situation is fixed for this one.
Ping me when it's the situation is settled!
Arthur
Comment 33 Victor Grousset/tuxayo 2022-09-18 17:42:44 UTC
David C in an email:

> I think we either need to revert the patches, or create new ones that remove the functionality, and then apply “Bug 31532 - Zebra search results incorrect because of Bug 15187” instead.

Having bug 31532 on top of this one don't solve the issue? Then is 31532 an alternative to this one? An incomplete alternative because «create new ones that remove the functionality, and then apply Bug 31532»
Comment 34 David Cook 2022-09-18 23:06:52 UTC
(In reply to Victor Grousset/tuxayo from comment #33)
> David C in an email:
> 
> > I think we either need to revert the patches, or create new ones that remove the functionality, and then apply “Bug 31532 - Zebra search results incorrect because of Bug 15187” instead.
> 
> Having bug 31532 on top of this one don't solve the issue? Then is 31532 an
> alternative to this one? An incomplete alternative because «create new ones
> that remove the functionality, and then apply Bug 31532»

Yeah, 31532 is an alternative to this one. 

My reasoning for not undoing 15187 via 31532 was to make it easy for 31532 to be backported to any version. I figure a maintainer can "git revert" this one rather than applying a new patch over top. If maintainers don't want to do that, I could provide another patch for 31532 to undo 15187.
Comment 35 Victor Grousset/tuxayo 2022-09-19 01:00:11 UTC
> Yeah, 31532 is an alternative to this one.

So in any case, this (bug 15187) should be reverted from main/master and stable?

> My reasoning for not undoing 15187 via 31532 was to make it easy for 31532 to be backported to any version. I figure a maintainer can "git revert" this one rather than applying a new patch over top. If maintainers don't want to do that, I could provide another patch for 31532 to undo 15187.

Yes, good idea.
Comment 36 David Cook 2022-09-19 05:35:39 UTC
(In reply to Victor Grousset/tuxayo from comment #35)
> > Yeah, 31532 is an alternative to this one.
> 
> So in any case, this (bug 15187) should be reverted from main/master and
> stable?
> 

Yep!
Comment 37 David Cook 2023-03-23 01:06:16 UTC
Looks like this wasn't reverted out of 22.05...
Comment 38 Katrin Fischer 2023-04-23 15:47:38 UTC
(In reply to David Cook from comment #37)
> Looks like this wasn't reverted out of 22.05...

Adding Rmaint and keyword.