Bug 33576 - Records are not indexed when imported if using Elasticsearch
Summary: Records are not indexed when imported if using Elasticsearch
Status: RESOLVED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Tools (show other bugs)
Version: Main
Hardware: All All
: P5 - low major (vote)
Assignee: Nick Clemens
QA Contact: Marcel de Rooy
URL:
Keywords:
: 35156 (view as bug list)
Depends on: 27421
Blocks:
  Show dependency treegraph
 
Reported: 2023-04-20 16:02 UTC by Nick Clemens
Modified: 2023-10-25 20:46 UTC (History)
4 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
23.05.00,22.11.06


Attachments
Bug 33576: Index records after import transaction is committed (1.57 KB, patch)
2023-04-20 16:05 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 33576: (QA follow-up) Polish comment, typo (1.67 KB, patch)
2023-04-24 09:30 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 33576: Index records after import transaction is committed (1.65 KB, patch)
2023-05-03 14:38 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 33576: (QA follow-up) Polish comment, typo (1.70 KB, patch)
2023-05-03 14:38 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 33576: Index records after import transaction is committed (1.74 KB, patch)
2023-05-09 13:01 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 33576: (QA follow-up) Polish comment, typo (1.74 KB, patch)
2023-05-09 13:02 UTC, Marcel de Rooy
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Nick Clemens 2023-04-20 16:02:03 UTC
The importing is done in a transaction. We call the indexer form inside the transaction. This means that when rabbit sends the task, it may not be found in the DB yet.

We need to ensure we index after the transaction completes
Comment 1 Nick Clemens 2023-04-20 16:05:15 UTC
Created attachment 149971 [details] [review]
Bug 33576: Index records after import transaction is committed

This patch simply moves our indexing call after the transaction is committed so
that the job will exist in the DB when called.

To test:
 1 - Have Koha using Elasticsearch
 2 - Stage and import a file of records
 3 - View the job in Admin->Manage jobs
 4 - Note it is not finished
 5 - Check log: /var/log/koha/kohadev/es-indexer-output.log
 6 - Note: [WARN] No job found for id=###
 7 - Apply patch
 8 - Stage and import
 9 - Note no error in log
10 - Note successful completion of indexing job
Comment 2 David Nind 2023-04-23 04:10:00 UTC
I'm successfully able to stage and import records and items (tried imports with and without items).

I'm not seeing the warning in the log.  However, I do have this error in /var/log/koha/kohadev/es-indexer-output.log - but this is there before doing any staging and importing: 

Cannot connect to the message broker, the jobs will be processed anyway (Failed to connect: Error connecting to localhost:61613: Cannot assign requested address at /usr/share/perl5/Net/Stomp.pm line 27.; giving up at /usr/share/perl5/Net/Stomp.pm line 27.
) at /kohadevbox/koha/misc/workers/es_indexer_daemon.pl line 84.

I've logged this as an issue for KTD https://gitlab.com/koha-community/koha-testing-docker/-/issues/373

Maybe this is why I can't replicate the issue for this bug?

Otherwise, is there anything else I need to be doing?
Comment 3 Marcel de Rooy 2023-04-24 09:30:18 UTC
Created attachment 150147 [details] [review]
Bug 33576: (QA follow-up) Polish comment, typo

No test plan.
Comment 4 Marcel de Rooy 2023-04-24 09:33:38 UTC
Feels like a race condition. So it depends who was faster: rabbitmq reading a new ticket or BatchCommitRecords doing the txn_commit.

Agree with this patch. Happy to QA it. Do not have Elastic at hand for testing though.
Comment 5 Jonathan Druart 2023-05-03 14:38:42 UTC
Created attachment 150591 [details] [review]
Bug 33576: Index records after import transaction is committed

This patch simply moves our indexing call after the transaction is committed so
that the job will exist in the DB when called.

To test:
 1 - Have Koha using Elasticsearch
 2 - Stage and import a file of records
 3 - View the job in Admin->Manage jobs
 4 - Note it is not finished
 5 - Check log: /var/log/koha/kohadev/es-indexer-output.log
 6 - Note: [WARN] No job found for id=###
 7 - Apply patch
 8 - Stage and import
 9 - Note no error in log
10 - Note successful completion of indexing job

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 6 Jonathan Druart 2023-05-03 14:38:44 UTC
Created attachment 150592 [details] [review]
Bug 33576: (QA follow-up) Polish comment, typo

No test plan.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 7 Marcel de Rooy 2023-05-09 13:01:59 UTC
Created attachment 150863 [details] [review]
Bug 33576: Index records after import transaction is committed

This patch simply moves our indexing call after the transaction is committed so
that the job will exist in the DB when called.

To test:
 1 - Have Koha using Elasticsearch
 2 - Stage and import a file of records
 3 - View the job in Admin->Manage jobs
 4 - Note it is not finished
 5 - Check log: /var/log/koha/kohadev/es-indexer-output.log
 6 - Note: [WARN] No job found for id=###
 7 - Apply patch
 8 - Stage and import
 9 - Note no error in log
10 - Note successful completion of indexing job

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 8 Marcel de Rooy 2023-05-09 13:02:02 UTC
Created attachment 150864 [details] [review]
Bug 33576: (QA follow-up) Polish comment, typo

No test plan.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 9 Tomás Cohen Arazi 2023-05-09 14:00:13 UTC
Pushed to master for 23.05.

Nice work everyone, thanks!
Comment 10 Matt Blenkinsop 2023-05-15 12:10:15 UTC
Nice work everyone!

Pushed to stable for 22.11.x
Comment 11 Lucas Gass 2023-06-15 16:48:06 UTC
Missing dependencies for 22.05.x, no backport.
Comment 12 Aleisha Amohia 2023-10-25 20:46:05 UTC
*** Bug 35156 has been marked as a duplicate of this bug. ***