Summary: | Adding 880 Fields to index-list in order to Increase Search for ALL non-latin Scripts | ||
---|---|---|---|
Product: | Koha | Reporter: | Jesse Lambertson <jesse> |
Component: | Searching | Assignee: | David Cook <dcook> |
Status: | CLOSED FIXED | QA Contact: | Martin Renvoize (ashimema) <martin.renvoize> |
Severity: | normal | ||
Priority: | P1 - high | CC: | arthur.suzuki, aspencatteam, barton, bgkriegel, david, dcook, flyingendpaper, J.P.Knight, lucas, martin.renvoize, mtompset, tomascohen, victor, ztajoli |
Version: | Main | Keywords: | rel_22_05_candidate |
Hardware: | All | ||
OS: | All | ||
Change sponsored?: | --- | Patch complexity: | Medium patch |
Documentation contact: | Documentation submission: | ||
Text to go in the release notes: |
This fixes the Zebra search engine when using ICU* so that 880 fields are rewritten as their linked fields and the alternate graphic representation of fields are indexed, in the same way that it works for Elasticsearch.
Example: add 245-01 to 880$6 and 教牧書信 to 880$a - the Chinese characters are now indexed into the title index using the 245 rules.
* ICU is a feature of the Zebra search engine that can be configured to make searching with non-latin languages (such as Chinese and Arabic) work correctly.
|
Version(s) released in: |
22.11.00, 22.05.05
|
Circulation function: | |||
Bug Depends on: | 31532 | ||
Bug Blocks: | 32033 | ||
Attachments: |
Zipped file of MARC records
MARC upload with a few examples with 880s Bug 15187: Index 880 in Zebra the same as Elasticsearch Bug 15187: Index 880 in Zebra the same as Elasticsearch Bug 15187: Index 880 in Zebra the same as Elasticsearch Bug 15187: (QA follow-up) Add unit test Bug 15187: Index 880 in Zebra the same as Elasticsearch Bug 15187: (QA follow-up) Add unit test |
Description
Jesse Lambertson
2015-11-13 15:23:57 UTC
Created attachment 44850 [details]
Zipped file of MARC records
Hi Jesse, may be I don't know which is the best solution to your problem, but as far as I understand 880 is not only to titles, but for any field that needs another graphical representation. I did a test with and Arabic title in a 246 field as parallel title, and I can found it from staff/opac as title without modifications. BTW, your zip file have a zero byte record. (In reply to Bernardo Gonzalez Kriegel from comment #2) > Hi Jesse, > may be I don't know which is the best solution to your problem, but as far > as I understand 880 is not only to titles, but for any field that needs > another graphical representation. > > I did a test with and Arabic title in a 246 field as parallel title, and I > can found it from staff/opac as title without modifications. > > BTW, your zip file have a zero byte record. I will reupload the MARC file then just for examples from our records. Thank you. Strange, Regarding the Arabic searching. You are correct that the issue affects all Arabic script in 880 fields. 245 and 246 will search because they are ALREADY indexed. 880s are not indexed separately from "keyword" searching so we can't search Arabic in any other way than in "keyword." Thank you, Jesse Created attachment 44876 [details]
MARC upload with a few examples with 880s
Faulty upload before, this file has a few records in it.
While this is needed for libraries with original script that use 880 for cataloguing, I think it's not a critical bug. Koha hasn't supported this so far, although it would be nice if it would. With DOM and the recently added options for conditional indexing it should be finally possible to make this work nicely. If this is not viewed as an important bug, can we drop the Importance "P1 - high" to be a lower category? (In reply to Jon Knight from comment #6) > If this is not viewed as an important bug, can we drop the Importance "P1 - > high" to be a lower category? Hi Jon, we are not using the Importance field at the moment. It would be ok to just change it. *** Bug 17407 has been marked as a duplicate of this bug. *** I'll be looking more at this soon. The solution I have in mind won't be as elegant as I would like, but it should be reasonably functional... Oh... actually this looks like it might be very difficult to do within the community model without making substantial changes to ./etc/zebradb/xsl/koha-indexdefs-to-zebra.xsl and ./etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml. These files were designed with too simple a logic in mind. I'm sorry but I don't have the resources to make those systematic architectural changes. I'm planning instead to locally customize ./etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl to have customized "match" attributes. For example: Replacing the following: <xslo:template mode="index_data_field" match="marc:datafield[@tag='245']"> With the following: <xslo:template mode="index_data_field" match="marc:datafield[@tag='245' or (@tag='880' and (marc:subfield[@code=6 and substring(text(),1,3)='245']))]"> While at a glance it's a very simple change, it's impossible to make that change using ./etc/zebradb/xsl/koha-indexdefs-to-zebra.xsl and ./etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml to create ./etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl I suppose you could customize ./etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml so that the 880 fields mirror the potential linkages (like 245), but it would be very easy for them to fall out of sync. My method of using a more complex "match" attribute is easier to maintain, although it does still run into the problem of needing to add the 880 logic for each possible mapping, which is labour-intensive and error-prone. Jesse, your original idea has merit, but it won't work with how the Koha Zebra indexing XSLT has been designed. Although I wonder if I can think of something else clever... (In reply to David Cook from comment #11) > Although I wonder if I can think of something else clever... Another thought is to do more transformations in the Zebra DOM pipeline... either by adding an "input" pipeline or by adding more "Extract" transform documents. (See https://software.indexdata.com/zebra/doc/idzebra.pdf) The idea would be that the 880 $6 245-01 would be copied into a 245 and that way it could just be fed through the normal indexing process. Another thing to think about is how this is Zebra-specific. I don't know what would need to be done for Elasticsearch (if anything)... It's a bit of a hack and I haven't tried it yet with the Zebra pipelines yet, but I'll include my transform here. It copies all the existing XML elements, but for 880 elements it creates a new element based off the linkage, and it just copies the indicators and subfields into that new element according to the MARC bib spec. I would've liked to have integrated this into Koha's existing XSLTs but I couldn't see an easy way of doing so, although I have a few ideas about how maybe it could work... <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:marc="http://www.loc.gov/MARC21/slim" version="1.0"> <xsl:output indent="yes" method="xml" version="1.0" encoding="UTF-8"/> <xsl:template name="identity" match="node()|@*"> <xsl:copy> <xsl:apply-templates select="@*|node()"/> </xsl:copy> </xsl:template> <xsl:template match="marc:datafield[@tag=880]"> <xsl:call-template name="identity"/> <xsl:variable name="tag" select="substring(marc:subfield[@code=6],1,3)"/> <xsl:text>
</xsl:text> <xsl:text> </xsl:text> <xsl:element name="datafield" namespace="http://www.loc.gov/MARC21/slim"> <xsl:attribute name="tag"><xsl:value-of select="substring(marc:subfield[@code=6],1,3)"/></xsl:attribute> <xsl:attribute name="ind1"><xsl:value-of select="@ind1"/></xsl:attribute> <xsl:attribute name="ind2"><xsl:value-of select="@ind2"/></xsl:attribute> <xsl:apply-templates select="marc:subfield[@code != '6']"/> </xsl:element> </xsl:template> </xsl:stylesheet> (In reply to David Cook from comment #13) > <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" > xmlns:marc="http://www.loc.gov/MARC21/slim" > version="1.0"> > <xsl:output indent="yes" > method="xml" > version="1.0" > encoding="UTF-8"/> > > <xsl:template name="identity" match="node()|@*"> > <xsl:copy> > <xsl:apply-templates select="@*|node()"/> > </xsl:copy> > </xsl:template> > > <xsl:template match="marc:datafield[@tag=880]"> > <xsl:call-template name="identity"/> > <xsl:variable name="tag" select="substring(marc:subfield[@code=6],1,3)"/> > <xsl:text>
</xsl:text> > <xsl:text> </xsl:text> > <xsl:element name="datafield" namespace="http://www.loc.gov/MARC21/slim"> > <xsl:attribute name="tag"><xsl:value-of > select="substring(marc:subfield[@code=6],1,3)"/></xsl:attribute> > <xsl:attribute name="ind1"><xsl:value-of > select="@ind1"/></xsl:attribute> > <xsl:attribute name="ind2"><xsl:value-of > select="@ind2"/></xsl:attribute> > <xsl:apply-templates select="marc:subfield[@code != '6']"/> > </xsl:element> > </xsl:template> > > </xsl:stylesheet> Steps to get this to work in a instance-specific context: 1. I saved the above as /etc/koha/zebradb/xsl/preprocess_marcxml.xsl 2. cp /etc/koha/zebradb/biblios/etc/dom-config.xml /etc/koha/sites/kohadev/test-dom-config.xml 3. vi /etc/koha/sites/kohadev/zebra-biblios-dom.cfg #For marcxml, I replaced /etc/koha/zebradb/biblios/etc/dom-config.xml with /etc/koha/sites/kohadev/test-dom-config.xml 4. vi /etc/koha/sites/kohadev/test-dom-config.xml #Use the following: <dom> <extract name="index"> <xslt stylesheet="preprocess_marcxml.xsl"/> <xslt stylesheet="biblio-zebra-indexdefs.xsl"/> </extract> <retrieve name="marc"> <xslt stylesheet="identity.xsl" /> </retrieve> <retrieve name="marcxml"> <xslt stylesheet="identity.xsl" /> </retrieve> <retrieve name="zebra"> <xslt stylesheet="zebra.xsl"/> </retrieve> <retrieve name="index"> <xslt stylesheet="biblio-zebra-indexdefs.xsl"/> </retrieve> <input> <xmlreader level="1"/> </input> </dom> 5. koha-rebuild-zebra -f -b -v kohadev The results are amazing! It works exactly like I hoped! I'm looking at a English/Chinese record that has many 880s for a variety of different fields. While Koha's default XSLTs will surface the 880 data on the detail page, *without this change*, the links don't work. However, with this change, they all work as you'd expect! Whether you click on the English or the Chinese, it's very smooth. -- Note that I haven't benchmarked this in a significant way. However, indexing 400 records in koha-testing-docker... there's no observable difference. By the way, it looks like our indexing code for Elasticsearch already handles 880s: my $tag = $field->tag(); # Handle alternate scripts in MARC 21 my $altscript = 0; if ($marcflavour eq 'marc21' && $tag eq '880') { my $sub6 = $field->subfield('6'); if ($sub6 =~ /^(...)-\d+/) { $tag = $1; $altscript = 1; } } In that case, it looks like they're not indexing 880 fields at all. Perhaps that makes sense since if you index the 880 and the linkage derived from it, you'd end up with 2x the keyword indexed fields... Of course, this transform is MARC21 specific, so it needs to be done in a way that is compatible with UNIMARC... (In reply to David Cook from comment #17) > Of course, this transform is MARC21 specific, so it needs to be done in a > way that is compatible with UNIMARC... I suppose another way to do this would be to change ./misc/migration_tools/rebuild_zebra.pl to use the same logic as ./Koha/SearchEngine/Elasticsearch.pm... Created attachment 134457 [details] [review] Bug 15187: Index 880 in Zebra the same as Elasticsearch This patch modifies the MARC21 export to Zebra, so that 880 fields are rewritten as their linked fields, in the same way that we already do with Elasticsearch, so that the alternate graphic representation of fields are indexed accordingly. (ie 880 $6245-01 Chinese titles will be indexed into the title index using the 245 rules) Test plan: 0. Apply patch 1. Turn on ICU indexing 1b. vi /etc/koha/zebradb/etc/default.idx 1c. Replace charmap word-phrase-utf.chr with icuchain words-icu.xml 1d. Replace charmap word-phrase-utf.chr with icuchain phrases-icu.xml 1e. Restart Zebra server 1f. Re-index Zebra 2. Add record with a 880 $6 245-01 $a 教牧書信 field. 3. Search for this record using a title index with the Chinese title 4. Note that the record is correctly retrieved (Note: This test probably works better using author or series as they present as links on the detail page which makes the fix more obviously useful.) Created attachment 137050 [details] [review] Bug 15187: Index 880 in Zebra the same as Elasticsearch This patch modifies the MARC21 export to Zebra, so that 880 fields are rewritten as their linked fields, in the same way that we already do with Elasticsearch, so that the alternate graphic representation of fields are indexed accordingly. (ie 880 $6245-01 Chinese titles will be indexed into the title index using the 245 rules) Test plan: 0. Apply patch 1. Turn on ICU indexing 1b. vi /etc/koha/zebradb/etc/default.idx 1c. Replace charmap word-phrase-utf.chr with icuchain words-icu.xml 1d. Replace charmap word-phrase-utf.chr with icuchain phrases-icu.xml 1e. Restart Zebra server 1f. Re-index Zebra 2. Add record with a 880 $6 245-01 $a 教牧書信 field. 3. Search for this record using a title index with the Chinese title 4. Note that the record is correctly retrieved (Note: This test probably works better using author or series as they present as links on the detail page which makes the fix more obviously useful.) Signed-off-by: David Nind <david@davidnind.com> Testing notes (using koha-testing-docker): 1. Setting up ICU: see comment 16 (notes 1 and 2) for bug 11158 2. Needed to update the bibliographic framework for books (BKS) to make 880$6 and 880$a visible in the editor. 3. Not related to this bug, but I did try to import the attached MARC records, but it stalled on Staged MARC record management screen, and still shows as importing (there are some other current bugs about issues with importing records). Error in the logs: [2022/07/03 21:25:48] [WARN] Filehandle STDOUT reopened as $fh only for input at /usr/share/perl5/DateTime/TimeZone/Local/Unix. pm line 155. 2 main::__ANON__ /kohadevbox/koha/tools/manage-marc-import.pl (331) 3 [2022/07/03 21:25:48] [WARN] DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::mysql::st execute failed: Cannot ad d or update a child row: a foreign key constraint fails (`koha_kohadev`.`items`, CONSTRAINT `items_ibfk_2` FOREIGN KEY (`homebr anch`) REFERENCES `branches` (`branchcode`) ON UPDATE CASCADE) at /kohadevbox/koha/Koha/Object.pm line 170 4 main::__ANON__ /kohadevbox/koha/tools/manage-marc-import.pl (331) 4. I had a go at the release notes text - feel free to edit! This is really nice and elegant David! QAing now. Created attachment 137207 [details] [review] Bug 15187: Index 880 in Zebra the same as Elasticsearch This patch modifies the MARC21 export to Zebra, so that 880 fields are rewritten as their linked fields, in the same way that we already do with Elasticsearch, so that the alternate graphic representation of fields are indexed accordingly. (ie 880 $6245-01 Chinese titles will be indexed into the title index using the 245 rules) Test plan: 0. Apply patch 1. Turn on ICU indexing 1b. vi /etc/koha/zebradb/etc/default.idx 1c. Replace charmap word-phrase-utf.chr with icuchain words-icu.xml 1d. Replace charmap word-phrase-utf.chr with icuchain phrases-icu.xml 1e. Restart Zebra server 1f. Re-index Zebra 2. Add record with a 880 $6 245-01 $a 教牧書信 field. 3. Search for this record using a title index with the Chinese title 4. Note that the record is correctly retrieved (Note: This test probably works better using author or series as they present as links on the detail page which makes the fix more obviously useful.) Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Created attachment 137208 [details] [review] Bug 15187: (QA follow-up) Add unit test This patch adds a unit test for the newly introduced filter. Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> I added a unit test for you to get this through QA.. Tomas may want to rename the filter at push.. I couldn't come up with a nicer name.. but I don't really like they we call out it's a Zebra thing... What I've not managed to test.. does this cause any issues with non-icu indexed Koha's.. I imagine not as you are unlikely to have such a field in your database in the first place without having enabled ICU (all our customers are ICU by default in case). Passing QA.. scripts are happy, we're now covered by tests and it's a great improvement. Created attachment 137233 [details] [review] Bug 15187: Index 880 in Zebra the same as Elasticsearch This patch modifies the MARC21 export to Zebra, so that 880 fields are rewritten as their linked fields, in the same way that we already do with Elasticsearch, so that the alternate graphic representation of fields are indexed accordingly. (ie 880 $6245-01 Chinese titles will be indexed into the title index using the 245 rules) Test plan: 0. Apply patch 1. Turn on ICU indexing 1b. vi /etc/koha/zebradb/etc/default.idx 1c. Replace charmap word-phrase-utf.chr with icuchain words-icu.xml 1d. Replace charmap word-phrase-utf.chr with icuchain phrases-icu.xml 1e. Restart Zebra server 1f. Re-index Zebra 2. Add record with a 880 $6 245-01 $a 教牧書信 field. 3. Search for this record using a title index with the Chinese title 4. Note that the record is correctly retrieved (Note: This test probably works better using author or series as they present as links on the detail page which makes the fix more obviously useful.) Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Created attachment 137234 [details] [review] Bug 15187: (QA follow-up) Add unit test This patch adds a unit test for the newly introduced filter. Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> I hate the filter name. But it is accurate! Pushed to master for 22.11. Nice work everyone, thanks! Backported to 22.05.x for 22.05.05 Ugh, this actually causes a regression with the processing of the 880 in Zebra search results, because we're fetching the MARCXML from Zebra instead of MySQL. I think we'll need to revert this change, and explore my Zebra-based one instead I think. Hi David, Given your last comment I won't backport to 21.11 until situation is fixed for this one. Ping me when it's the situation is settled! Arthur David C in an email: > I think we either need to revert the patches, or create new ones that remove the functionality, and then apply “Bug 31532 - Zebra search results incorrect because of Bug 15187” instead. Having bug 31532 on top of this one don't solve the issue? Then is 31532 an alternative to this one? An incomplete alternative because «create new ones that remove the functionality, and then apply Bug 31532» (In reply to Victor Grousset/tuxayo from comment #33) > David C in an email: > > > I think we either need to revert the patches, or create new ones that remove the functionality, and then apply “Bug 31532 - Zebra search results incorrect because of Bug 15187” instead. > > Having bug 31532 on top of this one don't solve the issue? Then is 31532 an > alternative to this one? An incomplete alternative because «create new ones > that remove the functionality, and then apply Bug 31532» Yeah, 31532 is an alternative to this one. My reasoning for not undoing 15187 via 31532 was to make it easy for 31532 to be backported to any version. I figure a maintainer can "git revert" this one rather than applying a new patch over top. If maintainers don't want to do that, I could provide another patch for 31532 to undo 15187. > Yeah, 31532 is an alternative to this one. So in any case, this (bug 15187) should be reverted from main/master and stable? > My reasoning for not undoing 15187 via 31532 was to make it easy for 31532 to be backported to any version. I figure a maintainer can "git revert" this one rather than applying a new patch over top. If maintainers don't want to do that, I could provide another patch for 31532 to undo 15187. Yes, good idea. (In reply to Victor Grousset/tuxayo from comment #35) > > Yeah, 31532 is an alternative to this one. > > So in any case, this (bug 15187) should be reverted from main/master and > stable? > Yep! Looks like this wasn't reverted out of 22.05... (In reply to David Cook from comment #37) > Looks like this wasn't reverted out of 22.05... Adding Rmaint and keyword. |