The importing is done in a transaction. We call the indexer form inside the transaction. This means that when rabbit sends the task, it may not be found in the DB yet. We need to ensure we index after the transaction completes
Created attachment 149971 [details] [review] Bug 33576: Index records after import transaction is committed This patch simply moves our indexing call after the transaction is committed so that the job will exist in the DB when called. To test: 1 - Have Koha using Elasticsearch 2 - Stage and import a file of records 3 - View the job in Admin->Manage jobs 4 - Note it is not finished 5 - Check log: /var/log/koha/kohadev/es-indexer-output.log 6 - Note: [WARN] No job found for id=### 7 - Apply patch 8 - Stage and import 9 - Note no error in log 10 - Note successful completion of indexing job
I'm successfully able to stage and import records and items (tried imports with and without items). I'm not seeing the warning in the log. However, I do have this error in /var/log/koha/kohadev/es-indexer-output.log - but this is there before doing any staging and importing: Cannot connect to the message broker, the jobs will be processed anyway (Failed to connect: Error connecting to localhost:61613: Cannot assign requested address at /usr/share/perl5/Net/Stomp.pm line 27.; giving up at /usr/share/perl5/Net/Stomp.pm line 27. ) at /kohadevbox/koha/misc/workers/es_indexer_daemon.pl line 84. I've logged this as an issue for KTD https://gitlab.com/koha-community/koha-testing-docker/-/issues/373 Maybe this is why I can't replicate the issue for this bug? Otherwise, is there anything else I need to be doing?
Created attachment 150147 [details] [review] Bug 33576: (QA follow-up) Polish comment, typo No test plan.
Feels like a race condition. So it depends who was faster: rabbitmq reading a new ticket or BatchCommitRecords doing the txn_commit. Agree with this patch. Happy to QA it. Do not have Elastic at hand for testing though.
Created attachment 150591 [details] [review] Bug 33576: Index records after import transaction is committed This patch simply moves our indexing call after the transaction is committed so that the job will exist in the DB when called. To test: 1 - Have Koha using Elasticsearch 2 - Stage and import a file of records 3 - View the job in Admin->Manage jobs 4 - Note it is not finished 5 - Check log: /var/log/koha/kohadev/es-indexer-output.log 6 - Note: [WARN] No job found for id=### 7 - Apply patch 8 - Stage and import 9 - Note no error in log 10 - Note successful completion of indexing job Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Created attachment 150592 [details] [review] Bug 33576: (QA follow-up) Polish comment, typo No test plan. Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Created attachment 150863 [details] [review] Bug 33576: Index records after import transaction is committed This patch simply moves our indexing call after the transaction is committed so that the job will exist in the DB when called. To test: 1 - Have Koha using Elasticsearch 2 - Stage and import a file of records 3 - View the job in Admin->Manage jobs 4 - Note it is not finished 5 - Check log: /var/log/koha/kohadev/es-indexer-output.log 6 - Note: [WARN] No job found for id=### 7 - Apply patch 8 - Stage and import 9 - Note no error in log 10 - Note successful completion of indexing job Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Created attachment 150864 [details] [review] Bug 33576: (QA follow-up) Polish comment, typo No test plan. Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Pushed to master for 23.05. Nice work everyone, thanks!
Nice work everyone! Pushed to stable for 22.11.x
Missing dependencies for 22.05.x, no backport.
*** Bug 35156 has been marked as a duplicate of this bug. ***