Bug 21286 - Advanced search for Corporate-name creates Zebra errors
Summary: Advanced search for Corporate-name creates Zebra errors
Status: RESOLVED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Zebra (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: Nick Clemens
QA Contact: Katrin Fischer
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-08-30 01:52 UTC by David Cook
Modified: 2021-07-10 23:39 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Text to go in the release notes:
This fixes the advanced search in the staff interface so that searching using the 'Corporate name' index now works correctly when the QueryAutoTruncate system preference is not enabled. Before this a search (using Zebra) for a name such as 'House plants' would not return any results and generate error messages in the log files.
Version(s) released in:
21.11.00,21.05.02,20.11.08


Attachments
Bug 21286: Add Corporate-name as phrase to zebra indexes (6.45 KB, patch)
2021-05-24 13:20 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 21286: Add Corporate-name as phrase to zebra indexes (6.48 KB, patch)
2021-06-02 21:36 UTC, David Nind
Details | Diff | Splinter Review
Bug 21286: Add Corporate-name as phrase to zebra indexes (6.55 KB, patch)
2021-06-03 09:56 UTC, Katrin Fischer
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description David Cook 2018-08-30 01:52:27 UTC
If you're in the advanced search in the staff client, and if you're using the QueryWeightFields system preference, and if you're searching with "Corporate-name", you'll generate Zebra errors.

There are a few things contributing to this problem. One is that Corporate-name is only indexed into the "word" register and not the "phrase" register. This becomes a problem with QueryWeightFields, because of the last conditional block in C4::Search::_build_weighted_query() which rewrites the search query to include $index,ext,r1, which forces a search of the "phrase" register, which causes Zebra to explode (when it doesn't use the 14=1 special attribute).
Comment 1 David Cook 2018-08-30 01:55:26 UTC
For example:

11:46:06-30/08 zebrasrv(44) [request] Search biblios ERROR 114 1 1+0 RPN @attrset Bib-1 @and @or @or @attr 1=Corporate-name @attr 4=1 @attr 6=3 @attr 9=32 @attr 2=102 corruption @attr 1=Corporate-name @attr 4=1 @attr 9=26 @attr 2=102 corruption @attr 1=Corporate-name @attr 4=6 @attr 9=20 @attr 2=102 corruption @attr 1=homebranch CPL

If you try that same search in yaz-client, you'll get something like:

Z> f @attrset Bib-1 @and @or @or @attr 1=Corporate-name @attr 4=1 @attr 6=3 @attr 9=32 @attr 2=102 corruption @attr 1=Corporate-name @attr 4=1 @attr 9=26 @attr 2=102 corruption @attr 1=Corporate-name @attr 4=6 @attr 9=20 @attr 2=102 corruption @attr 1=homebranch CPL
Sent searchRequest.
Received SearchResponse.
Search was a bloomin' failure.
Number of hits: 0, setno 1
Result Set Status: none
records returned: 0
Diagnostic message(s) from database:
    [114] Unsupported Use attribute -- v2 addinfo 'Corporate-name'
Elapsed: 0.006073

You can get some insight from this wiki entry: https://wiki.koha-community.org/wiki/Troubleshooting_Zebra#What_is_an_Unsupported_Use_attribute.3F (I wrote this years ago and it needs work but it's mostly on point.)

So it's an issue with the index not belonging in a particular register. As per my earlier comment and based on the search query, it's the "phrase" register it's missing from.

The interesting thing is that if we remove the @attr 6=3 from the above query, the query returns successfully. 

That's because the "phrase" register is only used for complete fields (ie for queries that have a @attr 6=3 attribute). You can see this in Table 5.13 at https://software.indexdata.com/zebra/doc/querymodel-zebra.html. 

Mind blown.
Comment 2 David Cook 2018-08-30 01:56:27 UTC
Now the question is do we put Corporate-name in the phrase register in the Zebra indexing XSLT...

...or do we re-think C4::Search::_build_weighted_query() because it's probably causing this exact same problem in lots of other contexts.
Comment 3 David Cook 2018-08-30 01:57:10 UTC
Or do we not care because we're just trying to move forward with Elasticsearch?
Comment 4 Fridolin Somers 2018-10-04 15:18:22 UTC
(In reply to David Cook from comment #2)
> Now the question is do we put Corporate-name in the phrase register in the
> Zebra indexing XSLT...
> 
> ...or do we re-think C4::Search::_build_weighted_query() because it's
> probably causing this exact same problem in lots of other contexts.

Indeed, its a tricky error. We have the same issue in UNIMARC.

I think we may change etc/zebradb/xsl/koha-indexdefs-to-zebra.xsl to always add phrase indexing on a words-list index (like Corporate-name:w)
Comment 5 Nick Clemens 2021-05-24 13:20:00 UTC
Created attachment 121326 [details] [review]
Bug 21286: Add Corporate-name as phrase to zebra indexes

When using Zebra for searching, Koha performs a number of searches in order
to improve relevancy. This means that even for 'wordlist' search, we perform a phrase search.

When selecting 'Corporate-name' as an index, this expansion of the search causes errors and fails
the search

We can fix this for 'Corporate-name' searches by adding a phrase index

To test:
 1 - Edit koha-conf.xml and uncomment the zebra debug line and add 'request' to the list
 2 - Restart all
 3 - tail -f /var/log/koha/kohadev/zebra-output.log
 4 - Edit a record to add a 110 field e.g. 'House plants'
 5 - Enable syspref IntranetCatalogSearchPulldown
 6 - Search for 'Corporate name' and term 'House plants'
 7 - No results
 8 - View the log, see 'ERROR' and full search terms listed
 9 - Apply patch
10 - copy the zebra files to the production instance:
    cp etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml /etc/koha/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml
    cp etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl /etc/koha/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl
11 - restart all
12 - rebuild: sudo koha-rebuild-zebra -v -f kohadev
13 - Repeat search
14 - Success!
Comment 6 David Nind 2021-05-30 01:33:25 UTC
I attempted to test this, but I couldn't recreate the error(steps 4-8) - I could find the record with the corporate name in it (I added a unique name not used anywhere else).

The QueryWeightFields system preference is enabled.

I changed the dropdown list to 'Corporate name' and also tried with 'Corporate name as a phrase'.

Is there something else I need to do to try and recreate the issue?

Here is the Zebra log entry for the request:

01:28:46-30/05 zebrasrv(54) [request] Search biblios OK 1 1 1+0 RPN @attrset Bib-1 @attr 1=Corporate-name @attr 4=6 @attr 5=1 @attr 2=102 "xanadu mftc"
01:28:46-30/05 zebrasrv(54) [request] Present OK -   1 1+1 
01:28:46-30/05 zebrasrv(54) [request] Present OK -   1 1+1 
01:28:46-30/05 zebrasrv(54) [request] Present OK -   1 1+1 
01:28:46-30/05 zebrasrv(54) [request] Present OK -   1 1+1 
01:28:46-30/05 zebrasrv(54) [request] Present OK -   1 1+1 
01:28:46-30/05 zebrasrv(54) [request] Present OK -   1 1+1 
01:28:46-30/05 zebrasrv(54) [request] Present OK -   1 1+1 
01:28:46-30/05 zebrasrv(54) [request] Present OK -   1 1+1 
01:28:46-30/05 zebrasrv(54) [request] Present OK -   1 1+1 
01:28:46-30/05 zebrasrv(54) [request] Present OK -   1 1+1 
01:28:46-30/05 zebrasrv(55) [request] Auth idPass kohauser -
01:28:46-30/05 zebrasrv(55) [request] Init OK - ID:81 Name:ZOOM-C/YAZ Version:5.27.1 872b6f92a024bb53bc1c11dfeccd47f065f98238
01:28:46-30/05 zebrasrv(55) [request] Search biblios OK 0 1 1+0 RPN @attrset Bib-1 @attr 1=1033 "Programming Perl /"


Testing notes (koha-testing-docker):
- Enabled Zebra logging:
  . edit /etc/koha/sites/kohadev/koha-conf.xml
  . find zebra_loglevels setting
  . uncomment and add request
- Added "xanadu mftc" as an authority record and linked to a record
Comment 7 Nick Clemens 2021-06-01 13:19:21 UTC
(In reply to David Nind from comment #6)
> I attempted to test this, but I couldn't recreate the error(steps 4-8) - I
> could find the record with the corporate name in it (I added a unique name
> not used anywhere else).
> 

Try disabling the syspref 'QueryAutoTruncate'  - that seems to be a key to triggering this
Comment 8 David Nind 2021-06-02 21:35:24 UTC
> Try disabling the syspref 'QueryAutoTruncate'  - that seems to be a key to
> triggering this

Thanks Nick! That worked, sign off on the way..
Comment 9 David Nind 2021-06-02 21:36:21 UTC
Created attachment 121554 [details] [review]
Bug 21286: Add Corporate-name as phrase to zebra indexes

When using Zebra for searching, Koha performs a number of searches in order
to improve relevancy. This means that even for 'wordlist' search, we perform a phrase search.

When selecting 'Corporate-name' as an index, this expansion of the search causes errors and fails
the search

We can fix this for 'Corporate-name' searches by adding a phrase index

To test:
 1 - Edit koha-conf.xml and uncomment the zebra debug line and add 'request' to the list
 2 - Restart all
 3 - tail -f /var/log/koha/kohadev/zebra-output.log
 4 - Edit a record to add a 110 field e.g. 'House plants'
 5 - Enable syspref IntranetCatalogSearchPulldown
 6 - Search for 'Corporate name' and term 'House plants'
 7 - No results
 8 - View the log, see 'ERROR' and full search terms listed
 9 - Apply patch
10 - copy the zebra files to the production instance:
    cp etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml /etc/koha/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml
    cp etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl /etc/koha/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl
11 - restart all
12 - rebuild: sudo koha-rebuild-zebra -v -f kohadev
13 - Repeat search
14 - Success!

Signed-off-by: David Nind <david@davidnind.com>
Comment 10 Katrin Fischer 2021-06-03 09:56:14 UTC
Created attachment 121567 [details] [review]
Bug 21286: Add Corporate-name as phrase to zebra indexes

When using Zebra for searching, Koha performs a number of searches in order
to improve relevancy. This means that even for 'wordlist' search, we perform a phrase search.

When selecting 'Corporate-name' as an index, this expansion of the search causes errors and fails
the search

We can fix this for 'Corporate-name' searches by adding a phrase index

To test:
 1 - Edit koha-conf.xml and uncomment the zebra debug line and add 'request' to the list
 2 - Restart all
 3 - tail -f /var/log/koha/kohadev/zebra-output.log
 4 - Edit a record to add a 110 field e.g. 'House plants'
 5 - Enable syspref IntranetCatalogSearchPulldown
 6 - Search for 'Corporate name' and term 'House plants'
 7 - No results
 8 - View the log, see 'ERROR' and full search terms listed
 9 - Apply patch
10 - copy the zebra files to the production instance:
    cp etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml /etc/koha/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml
    cp etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl /etc/koha/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl
11 - restart all
12 - rebuild: sudo koha-rebuild-zebra -v -f kohadev
13 - Repeat search
14 - Success!

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 11 Jonathan Druart 2021-06-21 10:05:11 UTC
Pushed to master for 21.11, thanks to everybody involved!
Comment 12 Kyle M Hall 2021-07-02 11:16:06 UTC
Pushed to 21.05.x for 21.05.02
Comment 13 Fridolin Somers 2021-07-10 02:19:37 UTC
Knowing that,
we may add in koha-indexdefs-to-zebra.xsl the phrase form if missing
Comment 14 Fridolin Somers 2021-07-10 02:20:29 UTC
Pushed to 20.11.x for 20.11.08
Comment 15 Victor Grousset/tuxayo 2021-07-10 23:39:53 UTC
Not backported to oldoldstable (20.05.x). Feel free to ask if it's needed.