When using bulkmarcimport for migrations it does make sense to do a full reindex after finishing the import. At the moment the import will always create entries in background_jobs for Elasticsearch and in zebra_queue for Zebra. We should have a switch to turn this off if indexing is not required to increase performance and avoid unnecessary load.
We need to adjust the calls to: AddBiblio ModBiblio ModBiblioMarc AddItemBatchFromMarc And add the skip_record_index parameter.
*** Bug 37565 has been marked as a duplicate of this bug. ***
Created attachment 170059 [details] [review] Bug 35466: Add skip_indexing parameter to bulkmarciport.pl This patch adds a new optoin to skip indexing to bulkmarcimport. It also fixes a bug where authorities were being indexed multiple times during import. To test: 1 - Apply patch 2 - Download the sample files on this bug 3 - Perform asearch engine search that should retrieve all records( 'a' in Zebra, '*' in ES) 4 - Note the total (435 in KTD) 5 - perl misc/migration_tools/bulkmarcimport.pl -b -v --file=bug_35466_b.mrc 6 - Search again, note increaed by 100 7 - perl misc/migration_tools/bulkmarcimport.pl -b -v --file=bug_35466_b.mrc --skip_indexing 8 - Search again, no increase 9 - perl misc/search_tools/rebuild_elasticsearch.pl -v 10 - Search again, increase, records were added but not initially indexed 11 - Browse to authorities and search as for biblios 12 - perl misc/migration_tools/bulkmarcimport.pl -a -v --file=bug_35466_a.mrc 13 - Search again, note increase 14 - perl misc/migration_tools/bulkmarcimport.pl -a -v --file=bug_35466_a.mrc --skip_indexing 15 - Search again, no increase 16 - perl misc/search_tools/rebuild_elasticsearch.pl -v 17 - Search again, increase, records were added but not initially indexed 18 - Sign off! Hi5!
Created attachment 170060 [details] Test auth file
Created attachment 170061 [details] Test biblios file
Test plan fails for me at step 13 while using ElasticSearch, there's no increase. Were authorities really being indexed multiple times? The way the indexing call is $indexer->update_index( \@search_engine_record_ids, \@search_engine_records ); and those params are only set for biblios, it looks to me from a quick skim of update_index() that for authorities we don't actually index if we don't index in ModAuthority/AddAuthority, because passing empty @record_ids and @records just indexes nothing.
Created attachment 170631 [details] [review] Bug 35466: (follow-up) Go to next if error, add to indexing call if not Before the previous patch we were indexing with every AddAuthority/ModAuthority call and I assumed we were also indexing during the commity, however, it appears we were not. This patch ensures we push the record and ids into the arrays for indexing during commit. Additionally I add a skip to next record on error, to match biblio behaviour. I also correct a log referring to biblios during authority importing
(In reply to Phil Ringnalda from comment #6) > Test plan fails for me at step 13 while using ElasticSearch, there's no > increase. > > Were authorities really being indexed multiple times? The way the indexing > call is $indexer->update_index( \@search_engine_record_ids, > \@search_engine_records ); and those params are only set for biblios, it > looks to me from a quick skim of update_index() that for authorities we > don't actually index if we don't index in ModAuthority/AddAuthority, because > passing empty @record_ids and @records just indexes nothing. Ah, I assumed the code was doing the right thing as well as the wrong thing, I was wrong ;-)
Created attachment 170641 [details] [review] Bug 35466: Add skip_indexing parameter to bulkmarciport.pl This patch adds a new optoin to skip indexing to bulkmarcimport. It also fixes a bug where authorities were being indexed multiple times during import. To test: 1 - Apply patch 2 - Download the sample files on this bug 3 - Perform asearch engine search that should retrieve all records( 'a' in Zebra, '*' in ES) 4 - Note the total (435 in KTD) 5 - perl misc/migration_tools/bulkmarcimport.pl -b -v --file=bug_35466_b.mrc 6 - Search again, note increaed by 100 7 - perl misc/migration_tools/bulkmarcimport.pl -b -v --file=bug_35466_b.mrc --skip_indexing 8 - Search again, no increase 9 - perl misc/search_tools/rebuild_elasticsearch.pl -v 10 - Search again, increase, records were added but not initially indexed 11 - Browse to authorities and search as for biblios 12 - perl misc/migration_tools/bulkmarcimport.pl -a -v --file=bug_35466_a.mrc 13 - Search again, note increase 14 - perl misc/migration_tools/bulkmarcimport.pl -a -v --file=bug_35466_a.mrc --skip_indexing 15 - Search again, no increase 16 - perl misc/search_tools/rebuild_elasticsearch.pl -v 17 - Search again, increase, records were added but not initially indexed 18 - Sign off! Hi5! Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Created attachment 170642 [details] [review] Bug 35466: (follow-up) Go to next if error, add to indexing call if not Before the previous patch we were indexing with every AddAuthority/ModAuthority call and I assumed we were also indexing during the commity, however, it appears we were not. This patch ensures we push the record and ids into the arrays for indexing during commit. Additionally I add a skip to next record on error, to match biblio behaviour. I also correct a log referring to biblios during authority importing Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Created attachment 172997 [details] [review] Bug 35466: (follow-up) document skip_indexing parameter to bulkmarciport.pl Add documentation for the new -si, --skip_indexing parameter
Created attachment 172998 [details] [review] Bug 35466: Add skip_indexing parameter to bulkmarciport.pl This patch adds a new optoin to skip indexing to bulkmarcimport. It also fixes a bug where authorities were being indexed multiple times during import. To test: 1 - Apply patch 2 - Download the sample files on this bug 3 - Perform asearch engine search that should retrieve all records( 'a' in Zebra, '*' in ES) 4 - Note the total (435 in KTD) 5 - perl misc/migration_tools/bulkmarcimport.pl -b -v --file=bug_35466_b.mrc 6 - Search again, note increaed by 100 7 - perl misc/migration_tools/bulkmarcimport.pl -b -v --file=bug_35466_b.mrc --skip_indexing 8 - Search again, no increase 9 - perl misc/search_tools/rebuild_elasticsearch.pl -v 10 - Search again, increase, records were added but not initially indexed 11 - Browse to authorities and search as for biblios 12 - perl misc/migration_tools/bulkmarcimport.pl -a -v --file=bug_35466_a.mrc 13 - Search again, note increase 14 - perl misc/migration_tools/bulkmarcimport.pl -a -v --file=bug_35466_a.mrc --skip_indexing 15 - Search again, no increase 16 - perl misc/search_tools/rebuild_elasticsearch.pl -v 17 - Search again, increase, records were added but not initially indexed 18 - Sign off! Hi5! Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org> Signed-off-by: Thomas Klausner <domm@plix.at>
Created attachment 172999 [details] [review] Bug 35466: (follow-up) Go to next if error, add to indexing call if not Before the previous patch we were indexing with every AddAuthority/ModAuthority call and I assumed we were also indexing during the commity, however, it appears we were not. This patch ensures we push the record and ids into the arrays for indexing during commit. Additionally I add a skip to next record on error, to match biblio behaviour. I also correct a log referring to biblios during authority importing Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org> Signed-off-by: Thomas Klausner <domm@plix.at>
Created attachment 173000 [details] [review] Bug 35466: (follow-up) document skip_indexing parameter to bulkmarciport.pl Add documentation for the new -si, --skip_indexing parameter Signed-off-by: Thomas Klausner <domm@plix.at>
Interestingly enough we needed exactly this feature today during a migration, so I QA'ed it and it works as advertised!
Pushed for 24.11! Well done everyone, thank you!
Fixed typo in bulkmarciport.pl in commit message on push.
Backported to 24.05.x for upcoming 24.05.06