Bug 36365 - compare-es-to-db tool should offer a mode to reconcile differences
Summary: compare-es-to-db tool should offer a mode to reconcile differences
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: Command-line Utilities (show other bugs)
Version: Main
Hardware: All All
: P3 enhancement
Assignee: Bugs List
QA Contact: Testopia
URL: https://bywatersolutions.com/crowdsou...
Keywords:
Depends on:
Blocks:
 
Reported: 2024-03-19 21:05 UTC by Liz Rea
Modified: 2024-09-27 14:48 UTC (History)
9 users (show)

See Also:
Change sponsored?: Seeking sponsor
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Liz Rea 2024-03-19 21:05:02 UTC
Koha can currently tell how many biblios are in the database vs how many are in the elasticsearch indexes, but it cannot reconcile those differences without either bespoke work deleting individual ES documents or a full reindex. A full reindex might not solve the problem anyway, due to invalid data in MARC records.

It would be really neat if the existing tool to compare-es-to-db.pl had a mode that would at least try to reconcile any discrepancies. For example if it was discovered that there were more indexes than biblios, remove the erroneous ES documents, or if there were more biblios than es indexes, identify them and attempt to reindex them.

We should also provide an easy to understand error report if reconciliation was not possible, with suggested actions to take for the sysadmins or users.

This would obviously be an Elasticsearch only feature.
Comment 1 Valerie 2024-07-09 15:41:01 UTC
This would be valuable for libraries--not only be able to see what the problem is but also have a first-line strategy for resolving it.
Comment 2 David Cook 2024-07-09 23:21:22 UTC
I like this idea.

When we were first getting started with Elasticsearch with Koha, we were having lots of issues, so we had to write to write a script to help fix up the discrepancies.

But it's difficult to do well. 

I think we need to start storing a timestamp in the Elasticsearch document, so that it can be compared against the biblio_metadata and items tables. If the timestamp for the bib or the item is greater than that of the ES document, then you could schedule a re-index (or maybe do a synchronous index since it's a CLI tool) for that particular record. 

Fortunately, we've managed to iron out the difficulties we were having, so this has been less of a pressing issue for us, but I still think it's a good idea for sure. The ES indexing can go wrong so easily, and it's currently so hard to repair without a full re-index :/.
Comment 3 George Williams (NEKLS) 2024-08-14 16:39:33 UTC
Good idea