Summary: | pseudonymize_statistics.pl script generates too many background jobs | ||
---|---|---|---|
Product: | Koha | Reporter: | Nick Clemens (kidclamp) <nick> |
Component: | Command-line Utilities | Assignee: | Nick Clemens (kidclamp) <nick> |
Status: | Pushed to main --- | QA Contact: | Marcel de Rooy <m.de.rooy> |
Severity: | normal | ||
Priority: | P5 - low | CC: | david, lisette, m.de.rooy, robin |
Version: | Main | ||
Hardware: | All | ||
OS: | All | ||
See Also: | https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39322 | ||
GIT URL: | Change sponsored?: | --- | |
Patch complexity: | Trivial patch | Documentation contact: | |
Documentation submission: | Text to go in the release notes: | ||
Version(s) released in: |
25.05.00
|
Circulation function: | |
Bug Depends on: | 34611 | ||
Bug Blocks: | 39322 | ||
Attachments: |
Bug 39301: pseudonymize_statistics.pl script generates too many background jobs
Bug 39301: pseudonymize_statistics.pl script generates too many background jobs Bug 39301: pseudonymize_statistics.pl script generates too many background jobs |
Description
Nick Clemens (kidclamp)
2025-03-11 11:10:31 UTC
Created attachment 179158 [details] [review] Bug 39301: pseudonymize_statistics.pl script generates too many background jobs This patch changes the script to use 'Koha::PseudonymizedTransaction->new_from_statistic' directly rather than calling 'Koha::Statistic->pseudonymize()' which generates a background job. Additionally, some of the checks from 'pseudonymnize' and the background job are moved into the search so we don't fetch any to pseudonymize that should not be. To test: 1 - Generate some entries in statistics (checkin, checkout, renew, etc) 2 - Enable pseudonymization (see bug 28911 if you need bcrypt settings) 3 - perl misc/maintenance/pseudonymize_statistics.pl -v -c 4 - Statistics pseudonymized 5 - Check the background jobs table, note jobs were generated 6 - Apply patch 7 - perl misc/maintenance/pseudonymize_statistics.pl -v -c 8 - You get same count as before, and are warned of existing pseudonymized transactions 9 - enter 'Y' to continue 10 - Check the pseudonymized_transacations table 11 - Confirm same rows were generated twice 12 - Check the background jobs 13 - Confirm there were no new jobs generated (for pseudonymization) Created attachment 179177 [details] [review] Bug 39301: pseudonymize_statistics.pl script generates too many background jobs This patch changes the script to use 'Koha::PseudonymizedTransaction->new_from_statistic' directly rather than calling 'Koha::Statistic->pseudonymize()' which generates a background job. Additionally, some of the checks from 'pseudonymnize' and the background job are moved into the search so we don't fetch any to pseudonymize that should not be. To test: 1 - Generate some entries in statistics (checkin, checkout, renew, etc) 2 - Enable pseudonymization (see bug 28911 if you need bcrypt settings) 3 - perl misc/maintenance/pseudonymize_statistics.pl -v -c 4 - Statistics pseudonymized 5 - Check the background jobs table, note jobs were generated 6 - Apply patch 7 - perl misc/maintenance/pseudonymize_statistics.pl -v -c 8 - You get same count as before, and are warned of existing pseudonymized transactions 9 - enter 'Y' to continue 10 - Check the pseudonymized_transacations table 11 - Confirm same rows were generated twice 12 - Check the background jobs 13 - Confirm there were no new jobs generated (for pseudonymization) Signed-off-by: David Nind <david@davidnind.com> Testing notes (using KTD): 1. Generate a password in KTD from the shell: htpasswd -bnBC 10 "" password | tr -d ':\n' | sed 's/$2y/$2a/' 2. Edit /etc/koha/sites/kohadev/koha-conf.xml 3. Add an entry before the </config>: <bcrypt_settings>whateverstep1generated</bcrypt_settings> 4. Restart everything: restart_all 5. Running the pseudonymize statistics script - if you get 0 statistics found, add the -b option with today's date, for example: perl misc/maintenance/pseudonymize_statistics.pl -b=2025-03-12 -v -c Created attachment 181155 [details] [review] Bug 39301: pseudonymize_statistics.pl script generates too many background jobs This patch changes the script to use 'Koha::PseudonymizedTransaction->new_from_statistic' directly rather than calling 'Koha::Statistic->pseudonymize()' which generates a background job. Additionally, some of the checks from 'pseudonymnize' and the background job are moved into the search so we don't fetch any to pseudonymize that should not be. To test: 1 - Generate some entries in statistics (checkin, checkout, renew, etc) 2 - Enable pseudonymization (see bug 28911 if you need bcrypt settings) 3 - perl misc/maintenance/pseudonymize_statistics.pl -v -c 4 - Statistics pseudonymized 5 - Check the background jobs table, note jobs were generated 6 - Apply patch 7 - perl misc/maintenance/pseudonymize_statistics.pl -v -c 8 - You get same count as before, and are warned of existing pseudonymized transactions 9 - enter 'Y' to continue 10 - Check the pseudonymized_transacations table 11 - Confirm same rows were generated twice 12 - Check the background jobs 13 - Confirm there were no new jobs generated (for pseudonymization) Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> QA comment: This is a pragmatic workaround, but not a complete solution. The actual cause in Koha/Statistic is not addressed here. Enqueuing one record for this process is a bad idea. Too much overhead. We should run it on a result set in the plural object. And probably not run it directly after adding to statistics but on a daily basis or so. Pushed for 25.05! Well done everyone, thank you! |