Bug 35815 - [Elasticsearch] Fix indexing of bibliographic records when items are deleted
Summary: [Elasticsearch] Fix indexing of bibliographic records when items are deleted
Status: Needs Signoff
Alias: None
Product: Koha
Classification: Unclassified
Component: Tools (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal (vote)
Assignee: Aleisha Amohia
QA Contact: Testopia
URL:
Keywords: Sandbox
Depends on:
Blocks:
 
Reported: 2024-01-16 02:24 UTC by Aleisha Amohia
Modified: 2024-03-28 14:29 UTC (History)
4 users (show)

See Also:
Change sponsored?: Sponsored
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 35815: Ensure ES biblio index is updated when items are batch deleted (5.16 KB, patch)
2024-01-16 02:34 UTC, Aleisha Amohia
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Aleisha Amohia 2024-01-16 02:24:32 UTC
Bibliographic records do not always get reindexed in Elasticsearch when items are batch deleted. This means that search results show an inaccurate number of remaining items.

To reproduce:

1. Spin up a Koha instance in your testing environment which uses Elasticsearch
2. Create a new bibliographic record and add three items to it
3. Do a search for your record that will return multiple results, so that you are shown the search results. Confirm that the search results shows three items are attached to the record.
4. Go to the record and select the checkbox for an item.
5. Click the link to Delete selected items. Confirm this again on the next page.
6. Confirm the job is enqueued then completed. It should say your item has been deleted.
7. Do a search for your record again so that you see it in the search results. Notice the number of items has not been updated - it still says three --> BUG
Comment 1 Aleisha Amohia 2024-01-16 02:34:49 UTC
Created attachment 161052 [details] [review]
Bug 35815: Ensure ES biblio index is updated when items are batch deleted

This patch moves the Elasticsearch indexing outside of the transaction. This ensures that bibliographic records are reindexed when items are batch deleted.

To test:

1. Spin up a Koha instance in your testing environment which uses Elasticsearch
2. Create a new bibliographic record and add three items to it
3. Do a search for your record that will return multiple results, so that you are shown the search results. Confirm that the search results shows three items are attached to the record.
4. Go to the record and select the checkbox for an item.
5. Click the link to Delete selected items. Confirm this again on the next page.
6. Confirm the job is enqueued then completed. It should say your item has been deleted.
7. Do a search for your record again so that you see it in the search results. Notice the number of items has not been updated - it still says three --> BUG

8. Apply the patch and restart services

9. Go to the record (there will be two items now) and select the checkbox for one item.
10. Click the link to Delete selected items. Confirm this again on the next page.
11. Confirm the job is enqueued then completed. It should say your item has been deleted.
12. Do a search for your record again so that you see it in the search results. Confirm the number of items HAS been updated. It should now show one item attached to the record.

Sponsored-by: Toi Ohomai Institute of Technology
Comment 2 David Nind 2024-01-17 09:22:42 UTC
I had a go at testing on the current master using koha-testing-docker (KTD).

I couldn't replicate the issue using Elasticsearch 7 and 8, and Open Search 1 and 2.

Happy to try with another Koha version, if you think that will make a difference.

Testing steps:
1. As per the test plan, added a record with title "DITA best practices" - minimal record with book as the item type.
2. Search was for "best" - 9 results with the sample data + new record.
Comment 3 Aleisha Amohia 2024-01-17 19:37:25 UTC
(In reply to David Nind from comment #2)
> I had a go at testing on the current master using koha-testing-docker (KTD).
> 
> I couldn't replicate the issue using Elasticsearch 7 and 8, and Open Search
> 1 and 2.
> 

That's a good point - it was definitely happening with ES 6, I'm not sure if it happened with ES 7. We might no longer be supporting ES 6 and since this patch was written we have actually upgraded to ES 7 locally anyway so it might not be needed... however I believe it is consistent with other ES indexing fixes I've seen recently in Koha.
Comment 4 David Nind 2024-01-17 21:47:11 UTC
I've tested using Koha 22.11 and ES6, and I couldn't replicate the issue.

Adding bug 35092 in case those working on the background job/worker issues have any comments.
Comment 5 Michaela Sieber 2024-02-19 15:26:43 UTC
There was a discussion on the devel mailinglist Wed, 31 Jan 2024 :

"We've had some reports that biblios which have been deleted via "Merge records" aren't always deleted from index and are shown in the search result list. When one opens such a result, Koha of course says there is no such a record in database."
Comment 6 Emmi Takkinen 2024-02-20 07:41:39 UTC
This has been problem for us for a while (using Koha 22.11 and Elasticsearch 6), mostly with combining biblios. But this occurs randomly and is hard to reproduce so I'm unable to provide any specific test case. We decided to wait and see if problem continues in new versions of Koha and Elasticsearch.
Comment 7 Clemens Tubach 2024-03-28 14:29:30 UTC
We also have this problem on production 23.05.04 using elastic 7.
The background job 'batch_item_record_deletion' was finished successfully and items are deleted.
But the related background job 'update_elastic_index' stays in status 'new', so the item is visible in the search result.

We couldn't reproduce it on current master with KTD either.

We hope that the problem will be solved with improvements in bug 35819 and related.