Bug 33576

Summary: Records are not indexed when imported if using Elasticsearch
Product: Koha Reporter: Nick Clemens (kidclamp) <nick>
Component: ToolsAssignee: Nick Clemens (kidclamp) <nick>
Status: CLOSED FIXED QA Contact: Marcel de Rooy <m.de.rooy>
Severity: major    
Priority: P5 - low CC: aleisha, david, lucas, m.de.rooy
Version: Main   
Hardware: All   
OS: All   
GIT URL: Change sponsored?: ---
Patch complexity: Small patch Documentation contact:
Documentation submission: Text to go in the release notes:
Version(s) released in:
23.05.00,22.11.06
Circulation function:
Bug Depends on: 27421    
Bug Blocks:    
Attachments: Bug 33576: Index records after import transaction is committed
Bug 33576: (QA follow-up) Polish comment, typo
Bug 33576: Index records after import transaction is committed
Bug 33576: (QA follow-up) Polish comment, typo
Bug 33576: Index records after import transaction is committed
Bug 33576: (QA follow-up) Polish comment, typo

Description Nick Clemens (kidclamp) 2023-04-20 16:02:03 UTC
The importing is done in a transaction. We call the indexer form inside the transaction. This means that when rabbit sends the task, it may not be found in the DB yet.

We need to ensure we index after the transaction completes
Comment 1 Nick Clemens (kidclamp) 2023-04-20 16:05:15 UTC
Created attachment 149971 [details] [review]
Bug 33576: Index records after import transaction is committed

This patch simply moves our indexing call after the transaction is committed so
that the job will exist in the DB when called.

To test:
 1 - Have Koha using Elasticsearch
 2 - Stage and import a file of records
 3 - View the job in Admin->Manage jobs
 4 - Note it is not finished
 5 - Check log: /var/log/koha/kohadev/es-indexer-output.log
 6 - Note: [WARN] No job found for id=###
 7 - Apply patch
 8 - Stage and import
 9 - Note no error in log
10 - Note successful completion of indexing job
Comment 2 David Nind 2023-04-23 04:10:00 UTC
I'm successfully able to stage and import records and items (tried imports with and without items).

I'm not seeing the warning in the log.  However, I do have this error in /var/log/koha/kohadev/es-indexer-output.log - but this is there before doing any staging and importing: 

Cannot connect to the message broker, the jobs will be processed anyway (Failed to connect: Error connecting to localhost:61613: Cannot assign requested address at /usr/share/perl5/Net/Stomp.pm line 27.; giving up at /usr/share/perl5/Net/Stomp.pm line 27.
) at /kohadevbox/koha/misc/workers/es_indexer_daemon.pl line 84.

I've logged this as an issue for KTD https://gitlab.com/koha-community/koha-testing-docker/-/issues/373

Maybe this is why I can't replicate the issue for this bug?

Otherwise, is there anything else I need to be doing?
Comment 3 Marcel de Rooy 2023-04-24 09:30:18 UTC
Created attachment 150147 [details] [review]
Bug 33576: (QA follow-up) Polish comment, typo

No test plan.
Comment 4 Marcel de Rooy 2023-04-24 09:33:38 UTC
Feels like a race condition. So it depends who was faster: rabbitmq reading a new ticket or BatchCommitRecords doing the txn_commit.

Agree with this patch. Happy to QA it. Do not have Elastic at hand for testing though.
Comment 5 Jonathan Druart 2023-05-03 14:38:42 UTC
Created attachment 150591 [details] [review]
Bug 33576: Index records after import transaction is committed

This patch simply moves our indexing call after the transaction is committed so
that the job will exist in the DB when called.

To test:
 1 - Have Koha using Elasticsearch
 2 - Stage and import a file of records
 3 - View the job in Admin->Manage jobs
 4 - Note it is not finished
 5 - Check log: /var/log/koha/kohadev/es-indexer-output.log
 6 - Note: [WARN] No job found for id=###
 7 - Apply patch
 8 - Stage and import
 9 - Note no error in log
10 - Note successful completion of indexing job

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 6 Jonathan Druart 2023-05-03 14:38:44 UTC
Created attachment 150592 [details] [review]
Bug 33576: (QA follow-up) Polish comment, typo

No test plan.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 7 Marcel de Rooy 2023-05-09 13:01:59 UTC
Created attachment 150863 [details] [review]
Bug 33576: Index records after import transaction is committed

This patch simply moves our indexing call after the transaction is committed so
that the job will exist in the DB when called.

To test:
 1 - Have Koha using Elasticsearch
 2 - Stage and import a file of records
 3 - View the job in Admin->Manage jobs
 4 - Note it is not finished
 5 - Check log: /var/log/koha/kohadev/es-indexer-output.log
 6 - Note: [WARN] No job found for id=###
 7 - Apply patch
 8 - Stage and import
 9 - Note no error in log
10 - Note successful completion of indexing job

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 8 Marcel de Rooy 2023-05-09 13:02:02 UTC
Created attachment 150864 [details] [review]
Bug 33576: (QA follow-up) Polish comment, typo

No test plan.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 9 Tomás Cohen Arazi (tcohen) 2023-05-09 14:00:13 UTC
Pushed to master for 23.05.

Nice work everyone, thanks!
Comment 10 Matt Blenkinsop 2023-05-15 12:10:15 UTC
Nice work everyone!

Pushed to stable for 22.11.x
Comment 11 Lucas Gass (lukeg) 2023-06-15 16:48:06 UTC
Missing dependencies for 22.05.x, no backport.
Comment 12 Aleisha Amohia 2023-10-25 20:46:05 UTC
*** Bug 35156 has been marked as a duplicate of this bug. ***