Koha can currently tell how many biblios are in the database vs how many are in the elasticsearch indexes, but it cannot reconcile those differences without either bespoke work deleting individual ES documents or a full reindex. A full reindex might not solve the problem anyway, due to invalid data in MARC records. It would be really neat if the existing tool to compare-es-to-db.pl had a mode that would at least try to reconcile any discrepancies. For example if it was discovered that there were more indexes than biblios, remove the erroneous ES documents, or if there were more biblios than es indexes, identify them and attempt to reindex them. We should also provide an easy to understand error report if reconciliation was not possible, with suggested actions to take for the sysadmins or users. This would obviously be an Elasticsearch only feature.
This would be valuable for libraries--not only be able to see what the problem is but also have a first-line strategy for resolving it.
I like this idea. When we were first getting started with Elasticsearch with Koha, we were having lots of issues, so we had to write to write a script to help fix up the discrepancies. But it's difficult to do well. I think we need to start storing a timestamp in the Elasticsearch document, so that it can be compared against the biblio_metadata and items tables. If the timestamp for the bib or the item is greater than that of the ES document, then you could schedule a re-index (or maybe do a synchronous index since it's a CLI tool) for that particular record. Fortunately, we've managed to iron out the difficulties we were having, so this has been less of a pressing issue for us, but I still think it's a good idea for sure. The ES indexing can go wrong so easily, and it's currently so hard to repair without a full re-index :/.
Good idea
Hi, I'm adding my two cents here. My first idea was to reuse the sub in about.pl for building a button in the about page, to trigger the fixes as a background job. The about page doesn't do the same thing as the `compare_es_to_db.pl` script, so it would take more time to write/add tests, etc. I focused on the CLI script this time, adding a convenient `--fix` option switch. Hope it helps.
Created attachment 178124 [details] [review] Bug 36365: Add a way for compare_es_to_db.pl to fix problems This patch adds a new option switch `--fix` (or `-f`) to make the script try to fix the problems it found. Bonus: `--help` option switch added. To test: 1. Apply this patch 2. Run: $ ktd --shell k$ perl -MKoha::SearchEngine -MKoha::SearchEngine::Indexer -e 'my $i = Koha::SearchEngine::Indexer->new({index => $Koha::SearchEngine::BIBLIOS_INDEX}); $i->delete_index([1]);' => SUCCESS: Bibliographic record #deleted from ES index 3. Run: k$ koha-mysql kohadev > DELETE FROM biblio WHERE biblionumber=2 OR biblionumber=3; => SUCCESS: Bibliographic records 2 and 3 deleted directly from the database, no reindex triggered by Koha. 4. Run: k$ perl misc/maintenance/compare_es_to_db.pl => SUCCESS: The results from (2) and (3) are explained. [1] 5. Run: k$ perl misc/maintenance/compare_es_to_db.pl --fix => SUCCESS: It says it is gonna fix things 6. Repeat 4 => SUCCESS: The problems we generated are no longer reported! They've been fixed! 7. Sign off :-D [1] Note KTD ships some broken records on purpose so devs need to deal with a not-perfect DB everyday, and catch misses in the code.
Created attachment 178178 [details] [review] Bug 36365: Add a way for compare_es_to_db.pl to fix problems This patch adds a new option switch `--fix` (or `-f`) to make the script try to fix the problems it found. Bonus: `--help` option switch added. To test: 1. Apply this patch 2. Run: $ ktd --shell k$ perl -MKoha::SearchEngine -MKoha::SearchEngine::Indexer -e 'my $i = Koha::SearchEngine::Indexer->new({index => $Koha::SearchEngine::BIBLIOS_INDEX}); $i->delete_index([1]);' => SUCCESS: Bibliographic record #deleted from ES index 3. Run: k$ koha-mysql kohadev > DELETE FROM biblio WHERE biblionumber=2 OR biblionumber=3; => SUCCESS: Bibliographic records 2 and 3 deleted directly from the database, no reindex triggered by Koha. 4. Run: k$ perl misc/maintenance/compare_es_to_db.pl => SUCCESS: The results from (2) and (3) are explained. [1] 5. Run: k$ perl misc/maintenance/compare_es_to_db.pl --fix => SUCCESS: It says it is gonna fix things 6. Repeat 4 => SUCCESS: The problems we generated are no longer reported! They've been fixed! 7. Sign off :-D [1] Note KTD ships some broken records on purpose so devs need to deal with a not-perfect DB everyday, and catch misses in the code. Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Created attachment 178874 [details] [review] Bug 36365: Add a way for compare_es_to_db.pl to fix problems Worked great and the --help is nice. This patch adds a new option switch `--fix` (or `-f`) to make the script try to fix the problems it found. Bonus: `--help` option switch added. To test: 1. Apply this patch 2. Run: $ ktd --shell k$ perl -MKoha::SearchEngine -MKoha::SearchEngine::Indexer -e 'my $i = Koha::SearchEngine::Indexer->new({index => $Koha::SearchEngine::BIBLIOS_INDEX}); $i->delete_index([1]);' => SUCCESS: Bibliographic record #deleted from ES index 3. Run: k$ koha-mysql kohadev > DELETE FROM biblio WHERE biblionumber=2 OR biblionumber=3; => SUCCESS: Bibliographic records 2 and 3 deleted directly from the database, no reindex triggered by Koha. 4. Run: k$ perl misc/maintenance/compare_es_to_db.pl => SUCCESS: The results from (2) and (3) are explained. [1] 5. Run: k$ perl misc/maintenance/compare_es_to_db.pl --fix => SUCCESS: It says it is gonna fix things 6. Repeat 4 => SUCCESS: The problems we generated are no longer reported! They've been fixed! 7. Sign off :-D [1] Note KTD ships some broken records on purpose so devs need to deal with a not-perfect DB everyday, and catch misses in the code. Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com> Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
All for the feature, but can we please get another set of QA eyes on this one? Thanks!
This is indexing each record individually - we should batch these up and use a method that implements background jobs - I fear this might overwhelm a server
(In reply to Nick Clemens (kidclamp) from comment #9) > This is indexing each record individually - we should batch these up and use > a method that implements background jobs - I fear this might overwhelm a > server Thanks, I will revisit this next week. My ultimate goal is to have something we can reuse from the UI later.
Created attachment 179623 [details] [review] Bug 36365: (follow-up) Indexing should be done async and in chunks Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
I'm happy to give this a second look, but to me it's looking great with the follow-up, do you agree Nick?
I think the patch works, but the ES records are not updated immediately. They end up in the `background_jobs` table. We had quite some problems with the `background_jobs`, and on ktd the missing record (id 1) is not added to ES. This might well be because I'm doing something wrong. In the DB the jobs have status 'finished', but ES is still missing: curl es:9200/koha_kohadev_biblios/data/1?pretty=true { "_index" : "koha_kohadev_biblios", "_type" : "data", "_id" : "1", "found" : false } Not sure what to do next...
I just did a manual reindex of the missing biblio (as I somehow don't get the background_job to actually index it, though according to the DB it did). So the patches work as advertised!
Created attachment 180724 [details] [review] Bug 36365: Add a way for compare_es_to_db.pl to fix problems This patch adds a new option switch `--fix` (or `-f`) to make the script try to fix the problems it found. Bonus: `--help` option switch added. To test: 1. Apply this patch 2. Run: $ ktd --shell k$ perl -MKoha::SearchEngine -MKoha::SearchEngine::Indexer -e 'my $i = Koha::SearchEngine::Indexer->new({index => $Koha::SearchEngine::BIBLIOS_INDEX}); $i->delete_index([1]);' => SUCCESS: Bibliographic record #deleted from ES index 3. Run: k$ koha-mysql kohadev > DELETE FROM biblio WHERE biblionumber=2 OR biblionumber=3; => SUCCESS: Bibliographic records 2 and 3 deleted directly from the database, no reindex triggered by Koha. 4. Run: k$ perl misc/maintenance/compare_es_to_db.pl => SUCCESS: The results from (2) and (3) are explained. [1] 5. Run: k$ perl misc/maintenance/compare_es_to_db.pl --fix => SUCCESS: It says it is gonna fix things 6. Repeat 4 => SUCCESS: The problems we generated are no longer reported! They've been fixed! 7. Sign off :-D [1] Note KTD ships some broken records on purpose so devs need to deal with a not-perfect DB everyday, and catch misses in the code. Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com> Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com> Signed-off-by: Thomas Klausner <domm@plix.at>
Created attachment 180725 [details] [review] Bug 36365: (follow-up) Indexing should be done async and in chunks Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: Thomas Klausner <domm@plix.at>
Pushed for 25.05! Well done everyone, thank you!
Nice work everyone! Pushed to 24.11.x for 24.11.04