In most parts of Koha, we use the MARCXML from the MySQL database, but in the search results we use the MARCXML from Zebra. While Bug 15187 allows us to index 880 according to its linkages, it means that the 880 is replaced and additional linked fields are created, which makes for some incorrect Zebra search results in Koha...
The solution from https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15187#c14 works well, especially for package installs. To overcome the issues from bug 15187, you can reindex the records at your leisure. If you don't overcome it immediately, everything will keep working. But when you next index a record, it'll have the best indexing done (in regards to the 880 handling). I'm implementing it locally, and I'll provide a patch here soon.
Created attachment 140331 [details] [review] Bug 31532: Add preprocessor to Zebra indexing pipeline to index 880 as linked field This patch adds a preprocessor XSLT to the Zebra indexing pipeline, so that 880 fields get indexed as the fields they're linked to. For example, a "880 $6 245" field would be indexed as a "245" field. However, because the preprocessor only occurs in the indexing part of the pipeline, it does not affect the retrieval of MARCXML from Zebra. That MARCXML will be the same MARCXML that was sent to Zebra from Koha. Test plan: 0. Revert bug 15187, and apply patch for 31532 1. cp ./etc/zebradb/biblios/etc/dom-config.xml /etc/koha/zebradb/biblios/etc/dom-config.xml 2a. cp etc/zebradb/marc_defs/marc21/biblios/preprocess_marcxml.xsl /etc/koha/zebradb/marc_defs/marc21/biblios/. 2b. cp etc/zebradb/marc_defs/normarc/biblios/preprocess_marcxml.xsl /etc/koha/zebradb/marc_defs/normarc/biblios/. 2c. cp etc/zebradb/marc_defs/unimarc/biblios/preprocess_marcxml.xsl /etc/koha/zebradb/marc_defs/unimarc/biblios/. 3. koha-rebuild-zebra -b -f -v kohadev 4. Note that in search results the 880$6245$a data appears before the 245$a data 5. Note that you can do a title index search on the 880$6245$a data
If you need 880 data to test with: 1. Go to http://localhost:8081/cgi-bin/koha/admin/marc_subfields_structure.pl?op=add_form&tagfield=880&frameworkcode= 2. Go to "a" and "6" subfields, and add "Editor" for "Visibility" 3. Go to http://localhost:8081/cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber=29&searchid=scs_1662620103145 4. Change framework to "Default framework" 5. In the "880" field, add "245-01" for the $6 6. In the "880" field, add "THISISATEST" for the $a (Using English for the 880$a makes it a bit easier for testing using CHR indexing.)
The revert bit means this can only be tested on a local installation - I'll give it a go.
* I verified the problem is only visible when using Zebra. * Found a search for my title returned all records on ktd = no ICU * Tried activating ICU: https://wiki.koha-community.org/wiki/ICU_chains_configuration * Noticed reindex failed: https://wiki.koha-community.org/wiki/ICU_chains_configuration * Killed the record with SQL * Reindexed again... Output is clean. * Search now brings back the exported record. * Why do we make this so hard? * Now revert: git revert 85690a14f8 git revert e6947b7408 git revert 058d411643 * Apply patch * Copy updated Zebra files so they are used: cp -i ./etc/zebradb/biblios/etc/dom-config.xml /etc/koha/zebradb/biblios/etc/dom-config.xml cp -i ./etc/zebradb/marc_defs/marc21/biblios/preprocess_marcxml.xsl /etc/koha/zebradb/marc_defs/marc21/biblios/ cp -i ./etc/zebradb/marc_defs/unimarc/biblios/preprocess_marcxml.xsl /etc/koha/zebradb/marc_defs/unimarc/biblios/ I've left out the change for NORMARC, as it's unsupported (see bug 18984) * Restarted Zebra server: sudo koha-zebra --restart -v kohadev 3. sudo koha-rebuild-zebra -b -f -v kohadev 4. Note that in search results the 880$6245$a data appears before the 245$a data 5. Note that you can do a title index search on the 880$6245$a data * Verify title words are not found in author index and vice versa :) All good, signing off!
Created attachment 140760 [details] [review] Bug 31532: Add preprocessor to Zebra indexing pipeline to index 880 as linked field This patch adds a preprocessor XSLT to the Zebra indexing pipeline, so that 880 fields get indexed as the fields they're linked to. For example, a "880 $6 245" field would be indexed as a "245" field. However, because the preprocessor only occurs in the indexing part of the pipeline, it does not affect the retrieval of MARCXML from Zebra. That MARCXML will be the same MARCXML that was sent to Zebra from Koha. Test plan: 0. Revert bug 15187, and apply patch for 31532 1. cp ./etc/zebradb/biblios/etc/dom-config.xml /etc/koha/zebradb/biblios/etc/dom-config.xml 2a. cp etc/zebradb/marc_defs/marc21/biblios/preprocess_marcxml.xsl /etc/koha/zebradb/marc_defs/marc21/biblios/. 2b. cp etc/zebradb/marc_defs/normarc/biblios/preprocess_marcxml.xsl /etc/koha/zebradb/marc_defs/normarc/biblios/. 2c. cp etc/zebradb/marc_defs/unimarc/biblios/preprocess_marcxml.xsl /etc/koha/zebradb/marc_defs/unimarc/biblios/. 3. koha-rebuild-zebra -b -f -v kohadev 4. Note that in search results the 880$6245$a data appears before the 245$a data 5. Note that you can do a title index search on the 880$6245$a data Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Created attachment 141337 [details] [review] Bug 31532: Add preprocessor to Zebra indexing pipeline to index 880 as linked field This patch adds a preprocessor XSLT to the Zebra indexing pipeline, so that 880 fields get indexed as the fields they're linked to. For example, a "880 $6 245" field would be indexed as a "245" field. However, because the preprocessor only occurs in the indexing part of the pipeline, it does not affect the retrieval of MARCXML from Zebra. That MARCXML will be the same MARCXML that was sent to Zebra from Koha. Test plan: 0. Revert bug 15187, and apply patch for 31532 1. cp ./etc/zebradb/biblios/etc/dom-config.xml /etc/koha/zebradb/biblios/etc/dom-config.xml 2a. cp etc/zebradb/marc_defs/marc21/biblios/preprocess_marcxml.xsl /etc/koha/zebradb/marc_defs/marc21/biblios/. 2b. cp etc/zebradb/marc_defs/normarc/biblios/preprocess_marcxml.xsl /etc/koha/zebradb/marc_defs/normarc/biblios/. 2c. cp etc/zebradb/marc_defs/unimarc/biblios/preprocess_marcxml.xsl /etc/koha/zebradb/marc_defs/unimarc/biblios/. 3. koha-rebuild-zebra -b -f -v kohadev 4. Note that in search results the 880$6245$a data appears before the 245$a data 5. Note that you can do a title index search on the 880$6245$a data Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Works well.. RM/RMaints remember to do the revert step in your branches as you roll this out. Passing QA. Note: I'm wondering now if this also affect Elastics as we effectively copied what we're doing there for bug 15187..
Please, add a comment about the required steps after upgrade for end users, so we add it to a technical section on the release notes.
Pushed to master for 22.11. Nice work everyone, thanks!
Backported to 22.05.x for upcoming 22.05.08
applied to 21.11 for 21.11.15
Not backported to 21.05.x
(In reply to Lucas Gass from comment #11) > Backported to 22.05.x for upcoming 22.05.08 It looks like bug 15187 wasn't reverted. I think that needs to be done too.