Bug 39301 - pseudonymize_statistics.pl script generates too many background jobs
Summary: pseudonymize_statistics.pl script generates too many background jobs
Status: Signed Off
Alias: None
Product: Koha
Classification: Unclassified
Component: Command-line Utilities (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal
Assignee: Nick Clemens (kidclamp)
QA Contact: Testopia
URL:
Keywords:
Depends on: 34611
Blocks:
  Show dependency treegraph
 
Reported: 2025-03-11 11:10 UTC by Nick Clemens (kidclamp)
Modified: 2025-03-13 13:15 UTC (History)
3 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments
Bug 39301: pseudonymize_statistics.pl script generates too many background jobs (2.60 KB, patch)
2025-03-11 11:24 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 39301: pseudonymize_statistics.pl script generates too many background jobs (2.64 KB, patch)
2025-03-11 19:11 UTC, David Nind
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Nick Clemens (kidclamp) 2025-03-11 11:10:31 UTC
The script works, but in a large system it generates a background job per statistic, this can choke the workers and slow system processing.

As this is intended as a one-time maintenance script I think we can remove the background-ing and pseudonymize the statistics directly
Comment 1 Nick Clemens (kidclamp) 2025-03-11 11:24:38 UTC
Created attachment 179158 [details] [review]
Bug 39301: pseudonymize_statistics.pl script generates too many background jobs

This patch changes the script to use 'Koha::PseudonymizedTransaction->new_from_statistic' directly
rather than calling 'Koha::Statistic->pseudonymize()' which generates a background job.

Additionally, some of the checks from 'pseudonymnize' and the background job are moved into the search
so we don't fetch any to pseudonymize that should not be.

To test:
 1 - Generate some entries in statistics (checkin, checkout, renew, etc)
 2 - Enable pseudonymization (see bug 28911 if you need bcrypt settings)
 3 - perl misc/maintenance/pseudonymize_statistics.pl -v -c
 4 - Statistics pseudonymized
 5 - Check the background jobs table, note jobs were generated
 6 - Apply patch
 7 - perl misc/maintenance/pseudonymize_statistics.pl -v -c
 8 - You get same count as before, and are warned of existing pseudonymized transactions
 9 - enter 'Y' to continue
10 - Check the pseudonymized_transacations table
11 - Confirm same rows were generated twice
12 - Check the background jobs
13 - Confirm there were no new jobs generated (for pseudonymization)
Comment 2 David Nind 2025-03-11 19:11:47 UTC
Created attachment 179177 [details] [review]
Bug 39301: pseudonymize_statistics.pl script generates too many background jobs

This patch changes the script to use 'Koha::PseudonymizedTransaction->new_from_statistic' directly
rather than calling 'Koha::Statistic->pseudonymize()' which generates a background job.

Additionally, some of the checks from 'pseudonymnize' and the background job are moved into the search
so we don't fetch any to pseudonymize that should not be.

To test:
 1 - Generate some entries in statistics (checkin, checkout, renew, etc)
 2 - Enable pseudonymization (see bug 28911 if you need bcrypt settings)
 3 - perl misc/maintenance/pseudonymize_statistics.pl -v -c
 4 - Statistics pseudonymized
 5 - Check the background jobs table, note jobs were generated
 6 - Apply patch
 7 - perl misc/maintenance/pseudonymize_statistics.pl -v -c
 8 - You get same count as before, and are warned of existing pseudonymized transactions
 9 - enter 'Y' to continue
10 - Check the pseudonymized_transacations table
11 - Confirm same rows were generated twice
12 - Check the background jobs
13 - Confirm there were no new jobs generated (for pseudonymization)

Signed-off-by: David Nind <david@davidnind.com>
Comment 3 David Nind 2025-03-11 19:18:06 UTC
Testing notes (using KTD):

1. Generate a password in KTD from the shell: htpasswd -bnBC 10 "" password | tr -d ':\n' | sed 's/$2y/$2a/'
2. Edit /etc/koha/sites/kohadev/koha-conf.xml
3. Add an entry before the </config>:
   <bcrypt_settings>whateverstep1generated</bcrypt_settings>
4. Restart everything: restart_all
5. Running the pseudonymize statistics script - if you get 0 statistics found, add the -b option with today's date, for example: perl misc/maintenance/pseudonymize_statistics.pl -b=2025-03-12 -v -c