Bug 33019 - Records not indexed in Elasticsearch ES when first catalogued
Summary: Records not indexed in Elasticsearch ES when first catalogued
Status: RESOLVED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: unspecified
Hardware: All All
: P5 - low blocker (vote)
Assignee: Nick Clemens
QA Contact:
URL:
Keywords: rel_22_11_candidate
Depends on: 30822
Blocks:
  Show dependency treegraph
 
Reported: 2023-02-21 03:37 UTC by Aleisha Amohia
Modified: 2023-10-25 20:43 UTC (History)
6 users (show)

See Also:
Change sponsored?: Sponsored
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
23.05.00,22.11.05


Attachments
Bug 33019: Make sure ES records are indexed the first time (1.02 KB, patch)
2023-02-21 03:40 UTC, Aleisha Amohia
Details | Diff | Splinter Review
Bug 33019: Make sure ES records are indexed the first time (1.79 KB, patch)
2023-03-21 12:40 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 33019: Make sure ES records are indexed the first time (2.47 KB, patch)
2023-03-21 13:04 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 33019: Make sure ES records are indexed the first time (2.53 KB, patch)
2023-03-22 20:53 UTC, Aleisha Amohia
Details | Diff | Splinter Review
Bug 33019: Make sure ES records are indexed the first time (2.60 KB, patch)
2023-03-23 09:02 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 33019: Make sure ES records are indexed the first time (2.60 KB, patch)
2023-03-23 09:03 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Aleisha Amohia 2023-02-21 03:37:23 UTC
We have a Koha instance where records are not being indexed into Elasticsearch when they are first catalogued. They can be accessed if you go to the catalogue detail page directly, but cannot be searched.

However if you edit the record and save, it is indexed successfully.
Comment 1 Aleisha Amohia 2023-02-21 03:39:25 UTC
We've come up with a hacky fix for this because I can't think of any other reason indexing isn't working, but I am really happy to be advised about this.
Comment 2 Aleisha Amohia 2023-02-21 03:40:39 UTC
Created attachment 146999 [details] [review]
Bug 33019: Make sure ES records are indexed the first time

This patch ensures records are indexed when they are created.
Comment 3 Katrin Fischer 2023-02-21 08:10:51 UTC
I have no idea if it's related yet, but there was also bug 33008 filed a little while ago. Could it be a similar issue?
Comment 4 Mark Hofstetter 2023-02-21 18:55:33 UTC
Hi Aleisha,

is the hack adding an item? if so it seems that it is related to https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33008

cheers
Comment 5 Aleisha Amohia 2023-02-21 21:25:41 UTC
Hi all,

Yes adding an item does cause the record to be indexed, but this Koha instance does not use items (they are a cataloguing agency).

Also it had nothing to do with Zebra being started or stopped, I tried cataloguing with Zebra on and with Zebra off and it had no effect.

Aleisha
Comment 6 Mark Hofstetter 2023-03-16 12:46:45 UTC
Hi,

I hope 

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33243

helps, you may use it in a cronjob 

eg every 5 minutes

rebuild_elasticsearch.pl -v -b --minutes 5

if you find in helpful and it works for you, please sign it off

thx
Mark
Comment 7 Nick Clemens 2023-03-21 12:40:19 UTC
Created attachment 148444 [details] [review]
Bug 33019: Make sure ES records are indexed the first time

This patch ensures records are indexed when they are created.
Previously, we were launching the indexer inside of of a transaction.
This meant that the job was being enqueued, but not being found by the
worker, becaue it was not yet in the DB

This patch skips record indexing in the transaction, and moves the indexing
command after

To test:
1 - Make sure you are using ES, and the es indexer is running
2 - tail -f /var/log/koha/kohadev/*.log
3 - Create a new record
4 - Note error in es-indexer-output.log like:
    [2023/03/21 12:22:36] [WARN] No job found for id=157 main:: /kohadevbox/koha/misc/workers/es_indexer_daemon.pl (129)
5 - Apply patch
6 - Create another record
7  There should be no error
8 - Search for the record and confirm it can be found
9 - View the background jobs in admin, confirm the most recent job has completed
Comment 8 Nick Clemens 2023-03-21 12:41:03 UTC
(In reply to Aleisha Amohia from comment #1)
> We've come up with a hacky fix for this because I can't think of any other
> reason indexing isn't working, but I am really happy to be advised about
> this.

I amended the patch, and added some comments, could you test Aleisha? It turns out the issue was that we were inside a transaction
Comment 9 Nick Clemens 2023-03-21 13:04:48 UTC
Created attachment 148447 [details] [review]
Bug 33019: Make sure ES records are indexed the first time

This patch ensures records are indexed when they are created.
Previously, we were launching the indexer inside of of a transaction.
This meant that the job was being enqueued, but not being found by the
worker, becaue it was not yet in the DB

This patch skips record indexing in the transaction, and moves the indexing
command after

To test:
1 - Make sure you are using ES, and the es indexer is running
2 - tail -f /var/log/koha/kohadev/*.log
3 - Create a new record
4 - Note error in es-indexer-output.log like:
    [2023/03/21 12:22:36] [WARN] No job found for id=157 main:: /kohadevbox/koha/misc/workers/es_indexer_daemon.pl (129)
5 - Apply patch
6 - Create another record
7  There should be no error
8 - Search for the record and confirm it can be found
9 - View the background jobs in admin, confirm the most recent job has completed
Comment 10 Aleisha Amohia 2023-03-22 20:53:05 UTC
Created attachment 148572 [details] [review]
Bug 33019: Make sure ES records are indexed the first time

This patch ensures records are indexed when they are created.
Previously, we were launching the indexer inside of of a transaction.
This meant that the job was being enqueued, but not being found by the
worker, becaue it was not yet in the DB

This patch skips record indexing in the transaction, and moves the indexing
command after

To test:
1 - Make sure you are using ES, and the es indexer is running
2 - tail -f /var/log/koha/kohadev/*.log
3 - Create a new record
4 - Note error in es-indexer-output.log like:
    [2023/03/21 12:22:36] [WARN] No job found for id=157 main:: /kohadevbox/koha/misc/workers/es_indexer_daemon.pl (129)
5 - Apply patch
6 - Create another record
7  There should be no error
8 - Search for the record and confirm it can be found
9 - View the background jobs in admin, confirm the most recent job has completed

Signed-off-by: Aleisha Amohia <aleishaamohia@hotmail.com>
Comment 11 Jonathan Druart 2023-03-23 09:02:17 UTC
Created attachment 148591 [details] [review]
Bug 33019: Make sure ES records are indexed the first time

This patch ensures records are indexed when they are created.
Previously, we were launching the indexer inside of of a transaction.
This meant that the job was being enqueued, but not being found by the
worker, becaue it was not yet in the DB

This patch skips record indexing in the transaction, and moves the indexing
command after

To test:
1 - Make sure you are using ES, and the es indexer is running
2 - tail -f /var/log/koha/kohadev/*.log
3 - Create a new record
4 - Note error in es-indexer-output.log like:
    [2023/03/21 12:22:36] [WARN] No job found for id=157 main:: /kohadevbox/koha/misc/workers/es_indexer_daemon.pl (129)
5 - Apply patch
6 - Create another record
7  There should be no error
8 - Search for the record and confirm it can be found
9 - View the background jobs in admin, confirm the most recent job has completed

Signed-off-by: Aleisha Amohia <aleishaamohia@hotmail.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 12 Jonathan Druart 2023-03-23 09:03:17 UTC
Created attachment 148592 [details] [review]
Bug 33019: Make sure ES records are indexed the first time

This patch ensures records are indexed when they are created.
Previously, we were launching the indexer inside of of a transaction.
This meant that the job was being enqueued, but not being found by the
worker, becaue it was not yet in the DB

This patch skips record indexing in the transaction, and moves the indexing
command after

To test:
1 - Make sure you are using ES, and the es indexer is running
2 - tail -f /var/log/koha/kohadev/*.log
3 - Create a new record
4 - Note error in es-indexer-output.log like:
    [2023/03/21 12:22:36] [WARN] No job found for id=157 main:: /kohadevbox/koha/misc/workers/es_indexer_daemon.pl (129)
5 - Apply patch
6 - Create another record
7  There should be no error
8 - Search for the record and confirm it can be found
9 - View the background jobs in admin, confirm the most recent job has completed

Signed-off-by: Aleisha Amohia <aleishaamohia@hotmail.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 13 Jonathan Druart 2023-03-23 09:04:22 UTC
I guess that's coming from bug 30822.
Comment 14 Tomás Cohen Arazi 2023-03-27 09:43:44 UTC
Pushed to master for 23.05.

Nice work everyone, thanks!
Comment 15 Matt Blenkinsop 2023-04-11 08:48:21 UTC
Nice work everyone!

Pushed to stable for 22.11.x
Comment 16 Lucas Gass 2023-04-24 16:56:16 UTC
Missing dependencies for 22.05.x, no backport.