Bug 25277

Summary: Z3950responder keyword search does not work with Elasticsearch 6
Product: Koha Reporter: Nick Clemens (kidclamp) <nick>
Component: Z39.50 / SRU / OpenSearch ServersAssignee: Nick Clemens (kidclamp) <nick>
Status: CLOSED FIXED QA Contact: Testopia <testopia>
Severity: major    
Priority: P5 - low CC: 1joynelson, black23, lucas, m.de.rooy, martin.renvoize, victor
Version: Main   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
20.05.00, 19.11.06
Circulation function:
Bug Depends on: 13937    
Bug Blocks: 25278    
Attachments: Bug 25277: Don't specify a field in query if none passed
Bug 25277: Unit test
Bug 25277: Don't specify a field in query if none passed
Bug 25277: Unit test
Bug 25277: Don't specify a field in query if none passed
Bug 25277: Unit test

Description Nick Clemens (kidclamp) 2020-04-24 13:32:41 UTC
The Z3950 responder is hardcoded to search the '_all" field if no field is passed. This breaks as of Koha 19.11 where the "_all" field is removed
Comment 1 Nick Clemens (kidclamp) 2020-04-24 13:34:50 UTC
Created attachment 103690 [details] [review]
Bug 25277: Don't specify a field in query if none passed
Comment 2 Nick Clemens (kidclamp) 2020-04-24 16:09:04 UTC
Created attachment 103698 [details] [review]
Bug 25277: Unit test
Comment 3 Victor Grousset/tuxayo 2020-04-26 22:25:03 UTC
IIUC , it's about using Koha as a Z39.50 server.
https://wiki.koha-community.org/wiki/Setting_up_the_Z39.50_and_SRU_Server

Is it valid to test it with the same instance of Koha as a client? ^^"
Comment 4 Nick Clemens (kidclamp) 2020-04-27 10:16:49 UTC
(In reply to Victor Grousset/tuxayo from comment #3)
> IIUC , it's about using Koha as a Z39.50 server.
> https://wiki.koha-community.org/wiki/Setting_up_the_Z39.50_and_SRU_Server
> 
> Is it valid to test it with the same instance of Koha as a client? ^^"

Yes, you can test on your test environment

To test:
0 - Have Koha running with ES enabled
1 - Launch the z3950 responder:
    perl misc/z3950_responder.pl -c etc/z3950/ --debug
2 - Connect using yaz-client:
    yaz-client localhost:2100
3 - base biblios
4 - find bruce
5 - No results
6 - Stop the responder
7 - Apply patch
8 - Repeat
9 - This time you get results

You can also setup the localhost as a z39 search and search using the cataloguing module
Comment 5 Victor Grousset/tuxayo 2020-05-04 10:00:19 UTC
For some reason I have issues to run the responder on koha-testing-docker

kohadev-koha@0c6291f3563f:/kohadevbox/koha$ perl misc/z3950_responder.pl -c etc/z3950/ --debug
/etc/koha/sites/kohadev/z3950/: No such file or directory

Why does it wants it from /etc/koha/sites/kohadev/z3950 ? It's like -c is ignored.

Same issue when using the koha-z3950-responder command.

kohadev-koha@0c6291f3563f:/kohadevbox/koha$ koha-z3950-responder --start kohadev
[....] Starting Z39.50/SRU daemon for kohadev:/etc/koha/sites/kohadev/z3950/: No such file or directory

I'm not specifying any param here. So it would be the default config of koha-testing-docker that doesn't pass it. Or it's ignored like above.

This ugly trick did it.
kohadev-koha@0c6291f3563f:/kohadevbox/koha$ ln -s /kohadevbox/koha/etc/z3950/ /etc/koha/sites/kohadev/z3950

Now it can try to test. But does anyone has an idea if that's and issue of Koha or koha-testing-docker?
Comment 6 Victor Grousset/tuxayo 2020-05-04 11:16:20 UTC
Z> find learning
Sent searchRequest.
Received SearchResponse.
Search was a success.
Number of hits: 4, setno 6
records returned: 0
Elapsed: 0.038088

It's the number of hits right?


Z> find learning
Sent searchRequest.
Received SearchResponse.
Search was a success.
Number of hits: 4, setno 1
records returned: 0
Elapsed: 0.048846

Same results with the patch and restart_all done.


> You can also setup the localhost as a z39 search and search using the cataloguing module

I got 3 results that way.

Wait. I still get 3 results without applying the patch. (I recreated my dev env)

Maybe there is something wrong with my ES setup? Is there another way to see the current issue?
Comment 7 Nick Clemens (kidclamp) 2020-05-04 11:26:32 UTC
(In reply to Victor Grousset/tuxayo from comment #6:
> Maybe there is something wrong with my ES setup? Is there another way to see
> the current issue?

Are you certain your SearchEngine syspref is set to elastic? I can still recreate the issue

Step 1 should say:
perl misc/z3950_responder.pl --config-dir /kohadevbox/koha/etc/z3950 --debug
Comment 8 Victor Grousset/tuxayo 2020-05-04 11:46:21 UTC
> Are you certain your SearchEngine syspref is set to elastic? I can still recreate the issue

Yes, I double checked that.

Z39.50 search gives me different results depending on the SearchEngine. But I always get results.
Comment 9 Victor Grousset/tuxayo 2020-05-04 14:01:19 UTC
Created attachment 104293 [details] [review]
Bug 25277: Don't specify a field in query if none passed

== test plan ==
To test:
0 - Have Koha running with ES 6 enabled
1 - Launch the z3950 responder:
    perl misc/z3950_responder.pl --config-dir /kohadevbox/koha/etc/z3950 --debug
2 - Connect using yaz-client:
    yaz-client localhost:2100
2.1 Alternative, if you add the responder as a Z39.50 server in Koha and
  do a catalogue Z39.50 search. Don't forget to search in the keywords
  field. Not the title field.
3 - base biblios
4 - find bruce
5 - No results
6 - Stop the responder
7 - Apply patch
8 - Repeat
9 - This time you get results

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 10 Victor Grousset/tuxayo 2020-05-04 14:01:24 UTC
Created attachment 104294 [details] [review]
Bug 25277: Unit test

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 11 Victor Grousset/tuxayo 2020-05-04 14:03:36 UTC
Test plan added. It was compiled from the previous comment. And with this addition:

> 2.1 Alternative, if you add the responder as a Z39.50 server in Koha and
>   do a catalogue Z39.50 search. Don't forget to search in the keywords
>   field. Not the title field.
Comment 12 Jonathan Druart 2020-05-15 13:17:42 UTC
Created attachment 104942 [details] [review]
Bug 25277: Don't specify a field in query if none passed

== test plan ==
To test:
0 - Have Koha running with ES 6 enabled
1 - Launch the z3950 responder:
    perl misc/z3950_responder.pl --config-dir /kohadevbox/koha/etc/z3950 --debug
2 - Connect using yaz-client:
    yaz-client localhost:2100
2.1 Alternative, if you add the responder as a Z39.50 server in Koha and
  do a catalogue Z39.50 search. Don't forget to search in the keywords
  field. Not the title field.
3 - base biblios
4 - find bruce
5 - No results
6 - Stop the responder
7 - Apply patch
8 - Repeat
9 - This time you get results

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Amended patch: Removed unecessary () in the return statement
Comment 13 Jonathan Druart 2020-05-15 13:17:46 UTC
Created attachment 104943 [details] [review]
Bug 25277: Unit test

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 14 Martin Renvoize (ashimema) 2020-05-18 14:29:06 UTC
Nice work everyone!

Pushed to master for 20.05
Comment 15 Joy Nelson 2020-05-19 23:05:54 UTC
backported to 19.11.x for 19.11.06
Comment 16 Lucas Gass (lukeg) 2020-05-22 14:39:05 UTC
missing dependencies for 19.05.x, no backport