Bug 31532 - Zebra search results including 880 with original script incorrect because of Bug 15187
Summary: Zebra search results including 880 with original script incorrect because of ...
Status: RESOLVED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Zebra (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: David Cook
QA Contact:
URL:
Keywords: release-notes-needed
Depends on:
Blocks: 15187
  Show dependency treegraph
 
Reported: 2022-09-08 04:34 UTC by David Cook
Modified: 2023-12-28 20:44 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
22.11.00, 22.05.08, 21.11.15


Attachments
Bug 31532: Add preprocessor to Zebra indexing pipeline to index 880 as linked field (5.41 KB, patch)
2022-09-08 06:53 UTC, David Cook
Details | Diff | Splinter Review
Bug 31532: Add preprocessor to Zebra indexing pipeline to index 880 as linked field (5.47 KB, patch)
2022-09-19 19:55 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 31532: Add preprocessor to Zebra indexing pipeline to index 880 as linked field (5.53 KB, patch)
2022-10-05 10:30 UTC, Martin Renvoize
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description David Cook 2022-09-08 04:34:04 UTC
In most parts of Koha, we use the MARCXML from the MySQL database, but in the search results we use the MARCXML from Zebra. 

While Bug 15187 allows us to index 880 according to its linkages, it means that the 880 is replaced and additional linked fields are created, which makes for some incorrect Zebra search results in Koha...
Comment 1 David Cook 2022-09-08 06:26:02 UTC
The solution from https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15187#c14 works well, especially for package installs. To overcome the issues from bug 15187, you can reindex the records at your leisure. If you don't overcome it immediately, everything will keep working. But when you next index a record, it'll have the best indexing done (in regards to the 880 handling).

I'm implementing it locally, and I'll provide a patch here soon.
Comment 2 David Cook 2022-09-08 06:53:27 UTC
Created attachment 140331 [details] [review]
Bug 31532: Add preprocessor to Zebra indexing pipeline to index 880 as linked field

This patch adds a preprocessor XSLT to the Zebra indexing pipeline,
so that 880 fields get indexed as the fields they're linked to. For example,
a "880 $6 245" field would be indexed as a "245" field.

However, because the preprocessor only occurs in the indexing part of the pipeline,
it does not affect the retrieval of MARCXML from Zebra. That MARCXML will be
the same MARCXML that was sent to Zebra from Koha.

Test plan:

0. Revert bug 15187, and apply patch for 31532
1. cp ./etc/zebradb/biblios/etc/dom-config.xml /etc/koha/zebradb/biblios/etc/dom-config.xml
2a. cp etc/zebradb/marc_defs/marc21/biblios/preprocess_marcxml.xsl /etc/koha/zebradb/marc_defs/marc21/biblios/.
2b. cp etc/zebradb/marc_defs/normarc/biblios/preprocess_marcxml.xsl /etc/koha/zebradb/marc_defs/normarc/biblios/.
2c. cp etc/zebradb/marc_defs/unimarc/biblios/preprocess_marcxml.xsl /etc/koha/zebradb/marc_defs/unimarc/biblios/.
3. koha-rebuild-zebra -b -f -v kohadev
4. Note that in search results the 880$6245$a data appears before the 245$a data
5. Note that you can do a title index search on the 880$6245$a data
Comment 3 David Cook 2022-09-08 07:01:43 UTC
If you need 880 data to test with:

1. Go to http://localhost:8081/cgi-bin/koha/admin/marc_subfields_structure.pl?op=add_form&tagfield=880&frameworkcode=
2. Go to "a" and "6" subfields, and add "Editor" for "Visibility"
3. Go to http://localhost:8081/cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber=29&searchid=scs_1662620103145
4. Change framework to "Default framework"
5. In the "880" field, add "245-01" for the $6
6. In the "880" field, add "THISISATEST" for the $a

(Using English for the 880$a makes it a bit easier for testing using CHR indexing.)
Comment 4 Katrin Fischer 2022-09-19 19:08:38 UTC
The revert bit means this can only be tested on a local installation - I'll give it a go.
Comment 5 Katrin Fischer 2022-09-19 19:54:05 UTC
* I verified the problem is only visible when using Zebra.
* Found a search for my title returned all records on ktd = no ICU
* Tried activating ICU: https://wiki.koha-community.org/wiki/ICU_chains_configuration
* Noticed reindex failed: https://wiki.koha-community.org/wiki/ICU_chains_configuration
* Killed the record with SQL
* Reindexed again... Output is clean.
* Search now brings back the exported record. 
* Why do we make this so hard?
* Now revert: 	
  git revert 85690a14f8
  git revert e6947b7408
  git revert 058d411643
* Apply patch
* Copy updated Zebra files so they are used:

cp -i ./etc/zebradb/biblios/etc/dom-config.xml /etc/koha/zebradb/biblios/etc/dom-config.xml
cp -i ./etc/zebradb/marc_defs/marc21/biblios/preprocess_marcxml.xsl /etc/koha/zebradb/marc_defs/marc21/biblios/
cp -i ./etc/zebradb/marc_defs/unimarc/biblios/preprocess_marcxml.xsl /etc/koha/zebradb/marc_defs/unimarc/biblios/

I've left out the change for NORMARC, as it's unsupported (see bug 18984)

* Restarted Zebra server: sudo koha-zebra --restart -v kohadev 
3. sudo koha-rebuild-zebra -b -f -v kohadev
4. Note that in search results the 880$6245$a data appears before the 245$a data
5. Note that you can do a title index search on the 880$6245$a data

* Verify title words are not found in author index and vice versa :)

All good, signing off!
Comment 6 Katrin Fischer 2022-09-19 19:55:48 UTC
Created attachment 140760 [details] [review]
Bug 31532: Add preprocessor to Zebra indexing pipeline to index 880 as linked field

This patch adds a preprocessor XSLT to the Zebra indexing pipeline,
so that 880 fields get indexed as the fields they're linked to. For example,
a "880 $6 245" field would be indexed as a "245" field.

However, because the preprocessor only occurs in the indexing part of the pipeline,
it does not affect the retrieval of MARCXML from Zebra. That MARCXML will be
the same MARCXML that was sent to Zebra from Koha.

Test plan:

0. Revert bug 15187, and apply patch for 31532
1. cp ./etc/zebradb/biblios/etc/dom-config.xml /etc/koha/zebradb/biblios/etc/dom-config.xml
2a. cp etc/zebradb/marc_defs/marc21/biblios/preprocess_marcxml.xsl /etc/koha/zebradb/marc_defs/marc21/biblios/.
2b. cp etc/zebradb/marc_defs/normarc/biblios/preprocess_marcxml.xsl /etc/koha/zebradb/marc_defs/normarc/biblios/.
2c. cp etc/zebradb/marc_defs/unimarc/biblios/preprocess_marcxml.xsl /etc/koha/zebradb/marc_defs/unimarc/biblios/.
3. koha-rebuild-zebra -b -f -v kohadev
4. Note that in search results the 880$6245$a data appears before the 245$a data
5. Note that you can do a title index search on the 880$6245$a data

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 7 Martin Renvoize 2022-10-05 10:30:37 UTC
Created attachment 141337 [details] [review]
Bug 31532: Add preprocessor to Zebra indexing pipeline to index 880 as linked field

This patch adds a preprocessor XSLT to the Zebra indexing pipeline,
so that 880 fields get indexed as the fields they're linked to. For example,
a "880 $6 245" field would be indexed as a "245" field.

However, because the preprocessor only occurs in the indexing part of the pipeline,
it does not affect the retrieval of MARCXML from Zebra. That MARCXML will be
the same MARCXML that was sent to Zebra from Koha.

Test plan:

0. Revert bug 15187, and apply patch for 31532
1. cp ./etc/zebradb/biblios/etc/dom-config.xml /etc/koha/zebradb/biblios/etc/dom-config.xml
2a. cp etc/zebradb/marc_defs/marc21/biblios/preprocess_marcxml.xsl /etc/koha/zebradb/marc_defs/marc21/biblios/.
2b. cp etc/zebradb/marc_defs/normarc/biblios/preprocess_marcxml.xsl /etc/koha/zebradb/marc_defs/normarc/biblios/.
2c. cp etc/zebradb/marc_defs/unimarc/biblios/preprocess_marcxml.xsl /etc/koha/zebradb/marc_defs/unimarc/biblios/.
3. koha-rebuild-zebra -b -f -v kohadev
4. Note that in search results the 880$6245$a data appears before the 245$a data
5. Note that you can do a title index search on the 880$6245$a data

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 8 Martin Renvoize 2022-10-05 10:32:09 UTC
Works well.. RM/RMaints remember to do the revert step in your branches as you roll this out.

Passing QA.

Note: I'm wondering now if this also affect Elastics as we effectively copied what we're doing there for bug 15187..
Comment 9 Tomás Cohen Arazi 2022-10-24 17:08:47 UTC
Please, add a comment about the required steps after upgrade for end users, so we add it to a technical section on the release notes.
Comment 10 Tomás Cohen Arazi 2022-10-24 17:41:30 UTC
Pushed to master for 22.11.

Nice work everyone, thanks!
Comment 11 Lucas Gass 2022-12-05 22:35:21 UTC
Backported to 22.05.x for upcoming 22.05.08
Comment 12 Arthur Suzuki 2022-12-16 15:39:58 UTC
applied to 21.11 for 21.11.15
Comment 13 wainuiwitikapark 2023-03-15 01:04:21 UTC
Not backported to 21.05.x
Comment 14 David Cook 2023-03-23 01:10:27 UTC
(In reply to Lucas Gass from comment #11)
> Backported to 22.05.x for upcoming 22.05.08

It looks like bug 15187 wasn't reverted. I think that needs to be done too.