Created attachment 89233 [details] [review] Bug 22831: Elasticsearch - add a maintenance script for checking DB vs index counts This script uses Array::Utils and adds a dependency To test: 1 - Have Koha with ES running 2 - Delete some records from ES curl -X DELETE "es:9200/koha_kohadev_biblios/data/5" curl -X DELETE "es:9200/koha_kohadev_authorities/data/5" 3 - perl misc/maintenance/compare_es_to_db.pl 4 - Note you are notified of problems in both indexes 5 - perl misc/search_tools/rebuild_elastic_search.pl -a 6 - perl misc/maintenance/compare_es_to_db.pl 7 - Note you are only notified about problems in biblios (assuming you don't have other issues) 8 - perl misc/search_tools/rebuild_elastic_search.pl -b 9 - perl misc/maintenance/compare_es_to_db.pl 10 - Both counts match, no problems
Created attachment 95077 [details] [review] Bug 22831: Elasticsearch - add a maintenance script for checking DB vs index counts This script uses Array::Utils and adds a dependency To test: 1 - Have Koha with ES running 2 - Delete some records from ES curl -X DELETE "es:9200/koha_kohadev_biblios/data/5" curl -X DELETE "es:9200/koha_kohadev_authorities/data/5" 3 - perl misc/maintenance/compare_es_to_db.pl 4 - Note you are notified of problems in both indexes 5 - perl misc/search_tools/rebuild_elastic_search.pl -a 6 - perl misc/maintenance/compare_es_to_db.pl 7 - Note you are only notified about problems in biblios (assuming you don't have other issues) 8 - perl misc/search_tools/rebuild_elastic_search.pl -b 9 - perl misc/maintenance/compare_es_to_db.pl 10 - Both counts match, no problems
I think this needs to display some sort of progress message. Otherwise it looks like it just hangs with high CPU usage. And please fix "elastic search" to "Elasticsearch" in the comment while at it. :) It would also be useful to check from which array a record is missing and tell it to the user. Since we already have the information in the arrays, let's help the user out.
Hi Nick, I troed to test it with Kohadevbox, but get this error on step 3: Can't locate Koha/Items.pm in @INC (you may need to install the Koha::Items module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.24.1 /usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24 /usr/share/perl/5.24 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at compare_es_to_db.pl line 33. BEGIN failed--compilation aborted at compare_es_to_db.pl line 33. I'm really not sure about this issue. It can be based at Kohadevbox instalation.
That's a weird one. The script worked for me in my dev environment.
Nick, one more thing: you can make the ES cursor a zillion times faster by not retrieving any of the stored fields: my $scroll = $es->scroll_helper( index => $searcher->get_elasticsearch_params->{index_name}, size => 5000, body => { query => { match_all => {} }, stored_fields => [] }, scroll_in_qs => 1, ); I'd also run the id comparison regardless of whether record counts match since there's could be a case where they do but the records still differ.
Created attachment 95105 [details] [review] Bug 22831: Elasticsearch - add a maintenance script for checking DB vs index counts This script uses Array::Utils and adds a dependency To test: 1 - Have Koha with ES running 2 - Delete some records from ES curl -X DELETE "es:9200/koha_kohadev_biblios/data/5" curl -X DELETE "es:9200/koha_kohadev_authorities/data/5" 3 - perl misc/maintenance/compare_es_to_db.pl 4 - Note you are notified of problems in both indexes 5 - perl misc/search_tools/rebuild_elastic_search.pl -a 6 - perl misc/maintenance/compare_es_to_db.pl 7 - Note you are only notified about problems in biblios (assuming you don't have other issues) 8 - perl misc/search_tools/rebuild_elasticsearch.pl -b 9 - perl misc/maintenance/compare_es_to_db.pl 10 - Both counts match, no problems
Nick, did you consider my suggestion about more informative result display: It would also be useful to check from which array a record is missing and tell it to the user. Since we already have the information in the arrays, let's help the user out. Also, maybe a running counter when fetching the records?
Created attachment 95109 [details] [review] Bug 22831: (follow-up) Determine where the record is missing and provide link, show counter
Created attachment 95110 [details] [review] Bug 22831: (follow-up) Determine where the record is missing and provide link, show counter
The url is a nice touch! A few issues remain, though: 1.) I'd still fetch the base urls outside the loop. Would improve readability and avoid repeated pref calls. 2.) For authorities, the OPAC URLs are now invalid. They should be something like [...]/cgi-bin/koha/opac-authoritiesdetail.pl?authid=123. 3.) The patch for PerlDependencies.pl is borked.
Very nice script, thanks a lot ;)
Created attachment 95154 [details] [review] Bug 22831: Elasticsearch - add a maintenance script for checking DB vs index counts This script uses Array::Utils and adds a dependency To test: 1 - Have Koha with ES running 2 - Delete some records from ES curl -X DELETE "es:9200/koha_kohadev_biblios/data/5" curl -X DELETE "es:9200/koha_kohadev_authorities/data/5" 3 - perl misc/maintenance/compare_es_to_db.pl 4 - Note you are notified of problems in both indexes 5 - perl misc/search_tools/rebuild_elastic_search.pl -a 6 - perl misc/maintenance/compare_es_to_db.pl 7 - Note you are only notified about problems in biblios (assuming you don't have other issues) 8 - perl misc/search_tools/rebuild_elasticsearch.pl -b 9 - perl misc/maintenance/compare_es_to_db.pl 10 - Both counts match, no problems
Created attachment 95155 [details] [review] Bug 22831: (follow-up) Determine where the record is missing and provide link, show counter
Created attachment 95156 [details] [review] Bug 22831: Elasticsearch - add a maintenance script for checking DB vs index counts This script uses Array::Utils and adds a dependency To test: 1 - Have Koha with ES running 2 - Delete some records from ES curl -X DELETE "es:9200/koha_kohadev_biblios/data/5" curl -X DELETE "es:9200/koha_kohadev_authorities/data/5" 3 - perl misc/maintenance/compare_es_to_db.pl 4 - Note you are notified of problems in both indexes 5 - perl misc/search_tools/rebuild_elastic_search.pl -a 6 - perl misc/maintenance/compare_es_to_db.pl 7 - Note you are only notified about problems in biblios (assuming you don't have other issues) 8 - perl misc/search_tools/rebuild_elasticsearch.pl -b 9 - perl misc/maintenance/compare_es_to_db.pl 10 - Both counts match, no problems Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Created attachment 95157 [details] [review] Bug 22831: (follow-up) Determine where the record is missing and provide link, show counter Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
record ids should be sorted numerically.
Created attachment 95176 [details] [review] Bug 22831: (follow-up) Sort results, group by db
Created attachment 96804 [details] [review] Bug 22831: Elasticsearch - add a maintenance script for checking DB vs index counts This script uses Array::Utils and adds a dependency To test: 1 - Have Koha with ES running 2 - Delete some records from ES curl -X DELETE "es:9200/koha_kohadev_biblios/data/5" curl -X DELETE "es:9200/koha_kohadev_authorities/data/5" 3 - perl misc/maintenance/compare_es_to_db.pl 4 - Note you are notified of problems in both indexes 5 - perl misc/search_tools/rebuild_elastic_search.pl -a 6 - perl misc/maintenance/compare_es_to_db.pl 7 - Note you are only notified about problems in biblios (assuming you don't have other issues) 8 - perl misc/search_tools/rebuild_elasticsearch.pl -b 9 - perl misc/maintenance/compare_es_to_db.pl 10 - Both counts match, no problems Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi> Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Created attachment 96805 [details] [review] Bug 22831: (follow-up) Determine where the record is missing and provide link, show counter Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi> Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Created attachment 96806 [details] [review] Bug 22831: (follow-up) Sort results, group by db Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Created attachment 97339 [details] [review] Bug 22831: Elasticsearch - add a maintenance script for checking DB vs index counts This script uses Array::Utils and adds a dependency To test: 1 - Have Koha with ES running 2 - Delete some records from ES curl -X DELETE "es:9200/koha_kohadev_biblios/data/5" curl -X DELETE "es:9200/koha_kohadev_authorities/data/5" 3 - perl misc/maintenance/compare_es_to_db.pl 4 - Note you are notified of problems in both indexes 5 - perl misc/search_tools/rebuild_elastic_search.pl -a 6 - perl misc/maintenance/compare_es_to_db.pl 7 - Note you are only notified about problems in biblios (assuming you don't have other issues) 8 - perl misc/search_tools/rebuild_elasticsearch.pl -b 9 - perl misc/maintenance/compare_es_to_db.pl 10 - Both counts match, no problems Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi> Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com> Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Created attachment 97340 [details] [review] Bug 22831: (follow-up) Determine where the record is missing and provide link, show counter Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi> Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com> Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Created attachment 97341 [details] [review] Bug 22831: (follow-up) Sort results, group by db Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com> Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
I would have preferred to not add the new deps, but not blocker. Thanks for the follow-up Nick!
Haha, I agree.. I would have preferred not to add a new dependancy... especially as it doesn't appear to be packaged for Jessie (having said that.. we are dropping supoprt for Jessie with 20.05 and this is an enhancement so that shouldn't be a problem). However.. I am wondering a little about the code having had a little read. You seem to do an array_diff and then two more iterations to catch where the diffs came from.. couldn't you have just done two array_minus calls (from the same module) to directly get the two different types of problem and saved yourself a loop and a few lines of code?
Created attachment 97783 [details] [review] Bug 22831: (RM follow-up) Code golf Rather than performing a symmetric diff and then splitting the results in a further loop this patch changes to logic to use two asymetric diffs to get the results more directly. Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Nice work everyone! Pushed to master for 20.05
Backported to 19.11.03
choosing not backport enhancement to 19.05.x
I wrote something like this for Zebra a long time ago, and while it is useful, it would be great to check if the contents of both the DB and the Index were the same. (For instance, the counts might be the same as you may not have added/deleted in a while, but the contents may still be different, if your updates aren't working.) What do people think about adding a checksum in a MARC field and indexing that? (Alternatively, I suppose we could just use the 005, although I don't know how reliably that is updated.)
*** Bug 17382 has been marked as a duplicate of this bug. ***