Bug 17871 - Can't retrieve facets (or zebra::snippet) from Zebra with YAZ 5.8.1
Summary: Can't retrieve facets (or zebra::snippet) from Zebra with YAZ 5.8.1
Alias: None
Product: Koha
Classification: Unclassified
Component: Z39.50 / SRU / OpenSearch Servers (show other bugs)
Version: master
Hardware: All All
: P5 - low major (vote)
Assignee: David Cook
QA Contact: Testopia
Keywords: Academy
Depends on:
Reported: 2017-01-10 01:03 UTC by David Cook
Modified: 2019-06-27 09:24 UTC (History)
6 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:

Bug 17871 - Can't retrieve facets (or zebra::snippet) from Zebra with YAZ 5.8.1 (984 bytes, patch)
2017-01-10 01:37 UTC, David Cook
Details | Diff | Splinter Review
Signed off patch (1.02 KB, patch)
2017-01-23 11:02 UTC, Colin Campbell
Details | Diff | Splinter Review
Bug 17871: Remove zebra::snippet to allow access to facets in YAZ 5.8.1+ (1.35 KB, patch)
2017-01-24 11:36 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 17871: (followup) Remove zebra::* from the packages templates (2.01 KB, patch)
2017-01-24 11:37 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description David Cook 2017-01-10 01:03:24 UTC
Colin Campbell and Frédéric Demians mentioned in 2015 that Zebra facets don't work with YAZ 5.8.1: https://lists.katipo.co.nz/public/koha/2015-February/041987.html

In 2016, Zeno said he couldn't get zebra::snippet to work: http://koha.1045719.n5.nabble.com/Zebra-the-elements-zebra-snippet-doesn-t-work-td5871444.html

I've looked at Indexdata's YAZ git repo for 5.8.1: https://github.com/indexdata/yaz/blame/v5.8.1/src/retrieval.c

And I think the problem is this commit:

At first, I thought the problem was with yaz_match_glob, but I've tested this function independently, and I think it's OK. I think the problem might be with how "el_best" is getting set. 

In theory, <retrieval syntax="xml"/> should be selected as el_best first... then zebra::* should be selected as it should have a true value returned from yaz_match_glob. (Note that "return 0" appears to indicate success for yaz_retrieval_request.)

So if you try to use facets with YAZ 5.8.1, you get the following from Zebra: 
[25] Specified element set name not valid for specified database -- v2 addinfo ''. 

That makes it seem like el_best isn't being set at all... but I don't know. I'm going to email Indexdata and see what they come up with for this one.

But in any case... I have a solution for now.

If we remove the line <retrieval syntax="xml" name="zebra::*" /> and restart Zebra, we can get results using zebra::facet and zebra::snippet. 

I've tested this change with OpenSuse Leap 42.2 with Zebra 2.0.60 and YAZ 5.8.1, and it works, but I'd be interested in hearing about other people's experiences with older (and newer) versions of YAZ. 

With commit 67abcc64, Marcel actually removed most unnecessary retrieval elements, but Tomas added them back with ca074c92 along with the zebra::* line.
Comment 1 David Cook 2017-01-10 01:35:43 UTC
I've sent emails to Adam Dickmeiss and info@indexdata.com, so hopefully IndexData can shine some light on this one.

In the meantime, I think a configuration change would be acceptable.
Comment 2 David Cook 2017-01-10 01:37:33 UTC
Created attachment 58718 [details] [review]
Bug 17871 - Can't retrieve facets (or zebra::snippet) from Zebra with YAZ 5.8.1

Remove line <retrieval syntax="xml" name="zebra::*" />, which causes
retrieval failures for zebra::* elements when using YAZ 5.8.1.
Comment 3 David Cook 2017-01-10 22:34:14 UTC
I've received a reply from Adam at IndexData, and he's included a fix in YAZ 5.20.0:

--- 5.20.0 2017/01/10

yaz-url: ignore content-length for HTTP method HEAD YAZ-878

retrieval: fix incorrect backend schema YAZ-877. Should fix

record_conv: allow HTTP method to be set for authority lookup
Comment 4 David Cook 2017-01-10 23:16:07 UTC
Here's the contents of the email that Adam sent me:

Hi David,

we've looked at this and it should be fixed as part of YAZ [5.20.0] which will be released shortly.

For a configuration like this:

<retrieval syntax="xml" name="zebra::*"/>

and using elem zebra::snippet the backend_schema i set to the name attribute zebra::* . The expected result is that the backend schema is set to the used element set zebra::snippet (Zebra issues error when getting zebra::*, but supports zebra::snippet).

For YAZ 5.8.0 and earlier, the first rule was matched .. For YAZ 5.8.1 the 2nd rule is matched when using zebra::snippet. So to some extend this error was hidden by the first rule, but the bug has probably always been there.

<retrieval syntax="xml"/>
<retrieval syntax="xml" name="zebra::*"/>


So in 5.8.0, the second rule wasn't being used anyway, so I think we're safe to remove it.
Comment 5 David Cook 2017-01-11 06:18:22 UTC
Adam from Indexdata has confirmed via email that we only need the <retrieval syntax="xml"/> line and that it will work with all YAZ versions.

So there should be no barriers to removing <retrieval syntax="xml" name="zebra::*"/>.
Comment 6 David Cook 2017-01-12 03:02:50 UTC
Found the original email I wrote about this last year:

Comment 7 David Cook 2017-01-22 22:55:06 UTC

#NOTE: You won't be able to test this using a Gitifid Koha, since it's an update to Zebra files. If you're using a Gitifid Koha, you'll have to apply the patch manually.

0) Install YAZ with a version >= 5.8.1 and < 5.20.0*. 

1) Build Koha from source using "make"
2) Ensure that Zebra facets are turned on in koha-conf.xml
3) Try to search in the OPAC, and note that no facets appear

4) Apply th patch
5) Build Koha from source using "Make"
6) Try to search in the OPAC, and note that facets now appear

*If you're using Indexdata's Debian repository, you'll need to pin the package so that you don't install the 5.20.0 where the software bug in YAZ itself is actually fixed, since we're testing configuration as a fix here.
Comment 8 Colin Campbell 2017-01-23 11:02:01 UTC
Created attachment 59400 [details] [review]
Signed off patch

Signed off - tested patch worked both against versions of yaz exhibiting the bug and an older release which did not exhibit it both provided facets with the config change in place
Comment 9 Colin Campbell 2017-01-23 11:02:52 UTC
Update status to Signed Off
Comment 10 David Cook 2017-01-24 02:04:11 UTC
You're a champion, Colin! Thanks!
Comment 11 Tomás Cohen Arazi 2017-01-24 11:36:13 UTC
Created attachment 59495 [details] [review]
Bug 17871: Remove zebra::snippet to allow access to facets in YAZ 5.8.1+

This patch restores access to zebra facets (or zebra::snippet) with YAZ 5.8.1 or higher.

It was failing due to The <retrieval syntax="xml" name="zebra::*" /> entry in
retrieval-info-bib-dom.xml which IndexData said it wasn't even needed to
get that access.

Edit: I amended the commit message (tcohen)

Signed-off-by: Colin Campbell <colin.campbell@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
I tested on kohadevbox and found no regression or behaviour change. I
will provide a followup for the packages.
Comment 12 Tomás Cohen Arazi 2017-01-24 11:37:08 UTC
Created attachment 59496 [details] [review]
Bug 17871: (followup) Remove zebra::* from the packages templates

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 13 Kyle M Hall 2017-02-07 17:50:40 UTC
Pushed to master for 17.05, thanks David, Tomas!
Comment 14 Katrin Fischer 2017-02-07 21:14:23 UTC
These patches have been pushed to 16.11.x and will be in 16.11.04.
Comment 15 Julian Maurice 2017-02-09 09:34:35 UTC
Pushed to 3.22.x for 3.22.17
Comment 16 Mason James 2017-02-22 23:07:59 UTC
Pushed to 16.05.x, for 16.05.10 release