Cleanup_database can delete all statistics entries more than X days old. If one is using pseudonymization to create pseudonymized_transactions data, then one may wish to use that cleanup_database function to delete statistics entries that have been duplicated in pseudonymized_transactions. However, not all types of transactions in statistics are duplicated in pseudonymized transactions. Statistics includes checkouts, returns, renewals, payments, writeoffs, and localuse. Pseudonymized_transactions only includes checkouts, returns, and renewals. It would be helpful to be able to delete only statistics entries more than X days old and in a specified list of types. Approaching this from the other end, Bug 32438 suggests adding payments and writeoffs to pseudonymized_transactions.
*** Bug 32439 has been marked as a duplicate of this bug. ***
Created attachment 172790 [details] [review] Bug 32440: Add missing unit test for Koha::Statistic->pseudonymize() To test: 1. prove t/db_dependent/Koha/Statistic.t Observe Subtest: pseudonymize() tests 1..6 ok 1 - some pseudonymization_types are defined ok 2 - Background job enqueued for type renew ok 3 - Background job enqueued for type issue ok 4 - Background job enqueued for type return ok 5 - Background job enqueued for type onsite_checkout ok 6 - Background job not enqueued for type not_gonna_pseudo ok 2 - pseudonymize() tests
Created attachment 172791 [details] [review] Bug 32440: Support purging statistics by type in cleanup_database.pl Cleanup_database can delete all statistics entries more than X days old. If one is using pseudonymization to create pseudonymized_transactions data, then one may wish to use that cleanup_database function to delete statistics entries that have been duplicated in pseudonymized_transactions. However, not all types of transactions in statistics are duplicated in pseudonymized transactions. Pseudonymized_transactions currently only includes checkouts, returns, and renewals. This patch adds two additional parameters to cleanup_database.pl: 1. --statistics-type Defines the types of statistics to purge. Will purge all types if parameter is omitted. Repeatable. 2. --statistics-type-pseudo Grabs values from @Koha::Statistic::pseudonymization_types. At the time of writing this patch, they are: renew, issue, return and onsite_checkout To test: 1. prove t/db_dependent/Koha/Statistic.t 2. Before applying this patch: 3. Create some statistics entry by checking out, renewing and checking in items. 4. perl misc/cronjobs/cleanup_database.pl --statistics 1 --verbose Observe: Purging statistics older than 1 days n statistics would have been removed Where n is the amount of statistics rows matching your test environment 5. Apply patch 6. Repeat step 4, observe same result 7. perl misc/cronjobs/cleanup_database.pl --statistics 1 --statistics-type-pseudo --verbose Observe: Purging statistics older than 1 days with types "onsite_checkout,renew,return,issue". n statistics would have been removed Where n is the amount of statistics rows matching your test environment (order of listed types does not matter) 8. perl misc/cronjobs/cleanup_database.pl --statistics 1 --statistics-type test1 --statistics-type test2 --verbose Observe: Purging statistics older than 1 days with types "test1,test2". n statistics would have been removed Where n is the amount of statistics rows matching your test environment (order of listed types does not matter) 9. perl misc/cronjobs/cleanup_database.pl --statistics 1 --statistics-type-pseudo --statistics-type test1 --verbose Observe: Purging statistics older than 1 days with types "test1,onsite_checkout,renew,return,issue". n statistics would have been removed Where n is the amount of statistics rows matching your test environment (order of listed types does not matter) 10. Try previous cleanup_database.pl commands with --confirm flag and make sure correct rows are purged
Created attachment 173874 [details] [review] Bug 32440: Add missing unit test for Koha::Statistic->pseudonymize() To test: 1. prove t/db_dependent/Koha/Statistic.t Observe Subtest: pseudonymize() tests 1..6 ok 1 - some pseudonymization_types are defined ok 2 - Background job enqueued for type renew ok 3 - Background job enqueued for type issue ok 4 - Background job enqueued for type return ok 5 - Background job enqueued for type onsite_checkout ok 6 - Background job not enqueued for type not_gonna_pseudo ok 2 - pseudonymize() tests Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Created attachment 173875 [details] [review] Bug 32440: Support purging statistics by type in cleanup_database.pl Cleanup_database can delete all statistics entries more than X days old. If one is using pseudonymization to create pseudonymized_transactions data, then one may wish to use that cleanup_database function to delete statistics entries that have been duplicated in pseudonymized_transactions. However, not all types of transactions in statistics are duplicated in pseudonymized transactions. Pseudonymized_transactions currently only includes checkouts, returns, and renewals. This patch adds two additional parameters to cleanup_database.pl: 1. --statistics-type Defines the types of statistics to purge. Will purge all types if parameter is omitted. Repeatable. 2. --statistics-type-pseudo Grabs values from @Koha::Statistic::pseudonymization_types. At the time of writing this patch, they are: renew, issue, return and onsite_checkout To test: 1. prove t/db_dependent/Koha/Statistic.t 2. Before applying this patch: 3. Create some statistics entry by checking out, renewing and checking in items. 4. perl misc/cronjobs/cleanup_database.pl --statistics 1 --verbose Observe: Purging statistics older than 1 days n statistics would have been removed Where n is the amount of statistics rows matching your test environment 5. Apply patch 6. Repeat step 4, observe same result 7. perl misc/cronjobs/cleanup_database.pl --statistics 1 --statistics-type-pseudo --verbose Observe: Purging statistics older than 1 days with types "onsite_checkout,renew,return,issue". n statistics would have been removed Where n is the amount of statistics rows matching your test environment (order of listed types does not matter) 8. perl misc/cronjobs/cleanup_database.pl --statistics 1 --statistics-type test1 --statistics-type test2 --verbose Observe: Purging statistics older than 1 days with types "test1,test2". n statistics would have been removed Where n is the amount of statistics rows matching your test environment (order of listed types does not matter) 9. perl misc/cronjobs/cleanup_database.pl --statistics 1 --statistics-type-pseudo --statistics-type test1 --verbose Observe: Purging statistics older than 1 days with types "test1,onsite_checkout,renew,return,issue". n statistics would have been removed Where n is the amount of statistics rows matching your test environment (order of listed types does not matter) 10. Try previous cleanup_database.pl commands with --confirm flag and make sure correct rows are purged Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>