Bug 18631

Summary: `cleanup_database.pl` should take an option for modules in action_logs
Product: Koha Reporter: Nick Clemens <nick>
Component: Command-line UtilitiesAssignee: Nick Clemens <nick>
Status: CLOSED FIXED QA Contact: Testopia <testopia>
Severity: enhancement    
Priority: P5 - low CC: andrewfh, barton, caroline.cyr-la-rose, jgaine, jonathan.druart, kyle, lisettepalouse+koha, martin.renvoize, rcoert, robin
Version: Main   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: ---
Documentation contact: Caroline Cyr La Rose Documentation submission: https://gitlab.com/koha-community/koha-manual/-/merge_requests/583
Text to go in the release notes:
This patch adds two new optional parameters to the `cleanup_database.pl` script. `--log-modules` - A repeatable option to specify which action log module lines to truncate. `--preserve-log` - A repeatable option to specify which action log module lines to keep.
Version(s) released in:
21.11.00
Attachments: Bug 18631: Add option to specify logs to preserve or delete in cleanup_database.pl
Bug 18631: Add option to specify logs to preserve or delete in cleanup_database.pl
Bug 18631: Add option to specify logs to preserve or delete in cleanup_database.pl
Bug 18631: Use - instead of _ for options

Description Nick Clemens 2017-05-18 18:17:48 UTC
Cataloging log is a big one that often needs to be cleaned, other logs can be useful to keep longer, we should take the modules as an option and clear only those specified or exclude specified logs
Comment 1 Andrew Fuerste-Henry 2021-03-02 18:14:05 UTC
+1 on this, libraries want some logs longer than others. Also, some logs are more feasible to keep longer. Granularity would be great.
Comment 2 Rebecca Coert 2021-03-04 21:33:13 UTC
100% agree!  It's possible that management is going to request a report that requires us to look back at important data that it more than 180 days old.  It's impossible to predict what information you'll need to access & when.  Recently ran into this issue looking at holds placed/filled/etc. in July 2020.  The report is empty because the data's been scrubbed by the cron.  

I second Andrew's request for granularity.
Comment 3 Jeff Gaines 2021-03-04 21:42:39 UTC
I would very much like more control over this setting. At the very least it should be a year's worth of data, but some smaller libraries might be able to hold multi-year logs. I support this request.
Comment 4 Nick Clemens 2021-07-27 18:50:39 UTC
Created attachment 123231 [details] [review]
Bug 18631: Add option to specify logs to preserve or delete in cleanup_database.pl

This adds two new parameters:
--log_module
--preserve_log

These can be repeated to include or exclude specific modules from the cleanup

To test:
0 - Apply patch
1 - Enable cataloging log and borrowers log
2 - Make some changes to borrowers and records
3 - run cleanup databse with --logs 0 --preserve_log=MEMBERS --preserve_log=CATALOGUING
4 - nothing is removed
5 - run cleanup databse with --logs 0 --log_module=MEMBERS
6 - the borrower logs are removed, the record changes remain
7 - run cleanup database with --logs 0
8 - record changes are removed (all other logs too)
Comment 5 Andrew Fuerste-Henry 2021-08-16 19:00:50 UTC
Created attachment 123857 [details] [review]
Bug 18631: Add option to specify logs to preserve or delete in cleanup_database.pl

This adds two new parameters:
--log_module
--preserve_log

These can be repeated to include or exclude specific modules from the cleanup

To test:
0 - Apply patch
1 - Enable cataloging log and borrowers log
2 - Make some changes to borrowers and records
3 - run cleanup databse with --logs 0 --preserve_log=MEMBERS --preserve_log=CATALOGUING
4 - nothing is removed
5 - run cleanup databse with --logs 0 --log_module=MEMBERS
6 - the borrower logs are removed, the record changes remain
7 - run cleanup database with --logs 0
8 - record changes are removed (all other logs too)

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>

Signed-off-by: Rebecca Coert <rcoert@arlingtonva.us>
Comment 6 Martin Renvoize 2021-09-30 10:45:13 UTC
Created attachment 125485 [details] [review]
Bug 18631: Add option to specify logs to preserve or delete in cleanup_database.pl

This adds two new parameters:
--log_module
--preserve_log

These can be repeated to include or exclude specific modules from the cleanup

To test:
0 - Apply patch
1 - Enable cataloging log and borrowers log
2 - Make some changes to borrowers and records
3 - run cleanup databse with --logs 0 --preserve_log=MEMBERS --preserve_log=CATALOGUING
4 - nothing is removed
5 - run cleanup databse with --logs 0 --log_module=MEMBERS
6 - the borrower logs are removed, the record changes remain
7 - run cleanup database with --logs 0
8 - record changes are removed (all other logs too)

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Rebecca Coert <rcoert@arlingtonva.us>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 7 Martin Renvoize 2021-09-30 10:46:18 UTC
Nice improvement... I fixed up the patch to drop the unintended Koha/XSLT/Base.pm debugging.. they were clearly an errant part of the commit. ;)

Passing QA
Comment 8 Jonathan Druart 2021-10-07 09:12:04 UTC
Created attachment 125864 [details] [review]
Bug 18631: Use - instead of _ for options

To match existing options
Comment 9 Jonathan Druart 2021-10-07 09:20:17 UTC
Pushed to master for 21.11, thanks to everybody involved!