Bug 21820

Summary: Zebraqueue should not be added to when only Elasticsearch is used
Product: Koha Reporter: Martin Renvoize (ashimema) <martin.renvoize>
Component: Searching - ElasticsearchAssignee: Peter Vashchuk <stalkernoid>
Status: Failed QA --- QA Contact:
Severity: enhancement    
Priority: P5 - low CC: dcook, didier.gautheron, fridolin.somers, nick, stalkernoid, wizzyrea
Version: Main   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26024
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:
Bug Depends on:    
Bug Blocks: 25710    
Attachments: Bug 21820: Don't add records to zebraqueue table when using Elasticsearch

Description Martin Renvoize (ashimema) 2018-11-13 13:27:28 UTC
In bug 20244 we identified that the zebraqueue is still being appended to when the searchengine preference is set to ElasticSearch.. unfortunately zebra may still be in use in parallel to Elastic to provide a public SRU/z3950 gateway.. as such we can't bind on just the syspref to know whether we should stop adding to the zebraqueue or not.
Comment 1 Katrin Fischer 2020-06-11 10:32:11 UTC
So I think maybe a new preference or option?
Comment 2 Fridolin Somers 2020-09-02 14:46:01 UTC
(In reply to Katrin Fischer from comment #1)
> So I think maybe a new preference or option?

Or a new value to SearchEngine : 
ES alone, ES + Zebra, Zebra alone
Comment 3 David Cook 2020-09-03 00:21:50 UTC
(In reply to Fridolin SOMERS from comment #2)
> (In reply to Katrin Fischer from comment #1)
> > So I think maybe a new preference or option?
> 
> Or a new value to SearchEngine : 
> ES alone, ES + Zebra, Zebra alone

That's probably more trouble than it's worth? 

I reckon a new system preference makes more sense?
Comment 4 Katrin Fischer 2020-10-18 16:58:06 UTC
I think a new option to the existing or a new pref would both work and be about the same amount of trouble :)
Comment 5 Fridolin Somers 2020-10-20 09:41:17 UTC
(In reply to Katrin Fischer from comment #4)
> I think a new option to the existing or a new pref would both work and be
> about the same amount of trouble :)

Thinking about it, actual pref "searchengine" is alread used.

A new pref whould be less disturbing.
Maybe EnableZebraQueue ?
Comment 6 Katrin Fischer 2022-02-19 23:46:30 UTC
*** Bug 29147 has been marked as a duplicate of this bug. ***
Comment 7 Katrin Fischer 2022-03-15 08:29:37 UTC
*** Bug 25710 has been marked as a duplicate of this bug. ***
Comment 8 Nick Clemens (kidclamp) 2023-10-17 20:45:55 UTC
333     #FIXME Current behaviour is to index Zebra when using ES, at some point we should stop
334     Koha::SearchEngine::Zebra::Indexer::index_records( $self, $record_numbers, $op, $server, undef );

Yeah, I think we are at the point :-)
Comment 9 Liz Rea 2023-10-17 20:46:53 UTC
let's make it a kconf flag, that way if you need it for whatever reason, a fallback is available
Comment 10 David Cook 2024-03-24 23:16:19 UTC
(In reply to Nick Clemens from comment #8)
> Yeah, I think we are at the point :-)

Agreed!

We're (finally) starting to use Elasticsearch more with Koha, and we're keen to disable Zebra completely for those sites using Elastic.

(In reply to Liz Rea from comment #9)
> let's make it a kconf flag, that way if you need it for whatever reason, a
> fallback is available

I like that idea. We can default it to Elasticsearch+Zebra for backwards compatibility, but have an option to not add to the zebraqueue. 

Another thing to consider is /etc/init.d/koha-common. We might want to add a --zebra option for koha-list, so that koha-zebra and koha-indexer are only invoked for Zebra-enabled instances.
Comment 11 Katrin Fischer 2024-08-16 11:29:05 UTC
*** Bug 37658 has been marked as a duplicate of this bug. ***
Comment 12 David Cook 2024-08-27 04:57:33 UTC
I'm hoping to look at this more later this year...
Comment 13 Peter Vashchuk 2024-11-14 10:24:18 UTC
Created attachment 174494 [details] [review]
Bug 21820: Don't add records to zebraqueue table when using Elasticsearch

Right now the index_records function in SearchEngine/Elasticsearch calls the index_records function from SearchEngine/Zebra, causing every items that we index for Elasticsearch to be also added to the zebraqueue table even when only Elasticsearch is in use. So this patch removes that call to prevent that from happening.

To reproduce:
1. While using Elasticsearch, edit any biblio record to activate the indexer.
2. Check zebraqueue table and confirm that new record was added.
3. Apply the patch.
4. Edit a biblio record again to trigger the indexer.
5. Check that it was not added to the zebraqueue table after you applied the patch.
6. Check that the biblio was indexed correctly: go to admin->jobs and verify that a new "Update Elasticsearch index" job has been added and completed, additionally use Elasticsearch to find the biblio record you edited to confirm that it works and nothing was broken.
Comment 14 David Cook 2024-11-14 22:31:20 UTC
(In reply to Martin Renvoize (ashimema) from comment #0)
> In bug 20244 we identified that the zebraqueue is still being appended to
> when the searchengine preference is set to ElasticSearch.. unfortunately
> zebra may still be in use in parallel to Elastic to provide a public
> SRU/z3950 gateway.. as such we can't bind on just the syspref to know
> whether we should stop adding to the zebraqueue or not.

Alternatively, Zebra might be used in cases where you want to provide a Z39.50 gateway that requires a password, as I don't think that z3950_responder.pl supports authentication.
Comment 15 David Cook 2024-11-14 22:33:03 UTC
This is on the right track, but it needs to have a configurable option so that people can continue to use both Elasticsearch and Zebra.