bulkmarcimport.pl disables syspref caching in order to explicitly set CataloguingLog and AuthoritiesLog to false (in order to boost performance, according to the notes). However, doing so results in numerous other syspref lookups over the course of the script, increasing runtime by about 33%. Sysprefs looked up during a plain, no-options import include: 1. DefaultClassificationSource 2. TimeFormat 3. dateformat 4. CataloguingLog 5. BiblioAddsAuthorities 6. marcflavour 7. SearchEngine 8. OAI-PMH:AutoUpdateSets While disabling additional logging on import is a good thing for performance, these additional DB queries offset that gain. Using the ENV variable override option may be a better way to temporarily disable these logs for the course of the script, whilst still keeping all the other sysprefs cached.
Created attachment 91898 [details] [review] Bug 23377: use ENV syspref override instead of complete disable In an attempt to save time, bulkmarcimport temporarily sets CataloguingLog and AuthoritiesLog to 0. It does this by disabling syspref caching and saving the changes to the database (then replacing the original values at completion). Unfortunately, this disables other key sysprefs from being cached, and results in a 50% increase in processing time for the script. This patch instead utilizes the ENV variable override feature of sysprefs, which preempts the cache in C4::Context->preference(). To test: 1. Perform a bulkmarcimport with a reasonable number of biblios (~1000 will do) 2. Note the time taken to complete 3. Apply patch 4. Revert the biblio load performed 5. Perform another bulkmarcimport with the same biblios and commandline options 6. Note the time taken to complete 7. Compare times. The time from step 6 should be about 33% less than the time from step 2 8. Check Cataloguing and Authorities Logs to verify imported records were not logged 9. Profit!
Created attachment 97143 [details] [review] Bug 23377: use ENV syspref override instead of complete disable In an attempt to save time, bulkmarcimport temporarily sets CataloguingLog and AuthoritiesLog to 0. It does this by disabling syspref caching and saving the changes to the database (then replacing the original values at completion). Unfortunately, this disables other key sysprefs from being cached, and results in a 50% increase in processing time for the script. This patch instead utilizes the ENV variable override feature of sysprefs, which preempts the cache in C4::Context->preference(). To test: 1. Perform a bulkmarcimport with a reasonable number of biblios (~1000 will do) 2. Note the time taken to complete 3. Apply patch 4. Revert the biblio load performed 5. Perform another bulkmarcimport with the same biblios and commandline options 6. Note the time taken to complete 7. Compare times. The time from step 6 should be about 33% less than the time from step 2 8. Check Cataloguing and Authorities Logs to verify imported records were not logged 9. Profit! Signed-off-by: Michal Denar <black23@gmail.com>
Looking here
Created attachment 97183 [details] [review] Bug 23377: use ENV syspref override instead of complete disable In an attempt to save time, bulkmarcimport temporarily sets CataloguingLog and AuthoritiesLog to 0. It does this by disabling syspref caching and saving the changes to the database (then replacing the original values at completion). Unfortunately, this disables other key sysprefs from being cached, and results in a 50% increase in processing time for the script. This patch instead utilizes the ENV variable override feature of sysprefs, which preempts the cache in C4::Context->preference(). To test: 1. Perform a bulkmarcimport with a reasonable number of biblios (~1000 will do) 2. Note the time taken to complete 3. Apply patch 4. Revert the biblio load performed 5. Perform another bulkmarcimport with the same biblios and commandline options 6. Note the time taken to complete 7. Compare times. The time from step 6 should be about 33% less than the time from step 2 8. Check Cataloguing and Authorities Logs to verify imported records were not logged 9. Profit! Signed-off-by: Michal Denar <black23@gmail.com> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Created attachment 97184 [details] [review] Bug 23377: (QA follow-up) Use OVERRIDE_SYSPREF SYSPREF_OVERRIDE unfortunately is not supported ;) Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Good to see you back, Ian. Ready to QA again :)
Nice work everyone! Pushed to master for 20.05
Pushed to 19.11.x branch for 19.11.03
backported to 19.05.x for 19.05.08
Enhancement will not be backported to 18.11.x series.