Bug 30069

Summary: Add edifact_messages to cleanup_database.pl
Product: Koha Reporter: Martin Renvoize (ashimema) <martin.renvoize>
Component: Command-line UtilitiesAssignee: Matt Blenkinsop <matt.blenkinsop>
Status: RESOLVED FIXED QA Contact: Kyle M Hall (khall) <kyle>
Severity: enhancement    
Priority: P5 - low CC: aude.charillon, fiona.borthwick, kyle, matt.blenkinsop, nick, robin
Version: Main   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30070
Change sponsored?: --- Patch complexity: Trivial patch
Documentation contact: Aude Charillon Documentation submission: https://gitlab.com/koha-community/koha-manual/-/merge_requests/947
Text to go in the release notes:
Version(s) released in:
23.05.00
Circulation function:
Attachments: Bug 30069: Add edifact-messages to cleanup_database.pl
Bug 30069: Add edifact-messages to cleanup_database.pl
Bug 30069: Add edifact-messages to cleanup_database.pl
Bug 30069: (QA follow-up) Fix capitalization for edifact
Bug 30069: (QA follow-up) Rebase and add filter
Bug 30069: Add edifact-messages to cleanup_database.pl
Bug 30069: (QA follow-up) Fix capitalization for edifact
Bug 30069: (QA follow-up) Rebase and add filter
Bug 30069: (QA follow-up) Use pure DBIC for select and delete with single query

Description Martin Renvoize (ashimema) 2022-02-10 13:18:51 UTC
We should add the ability to clean up old edifact messages with cleanup database.
Comment 1 Matt Blenkinsop 2023-02-15 14:26:05 UTC
Created attachment 146690 [details] [review]
Bug 30069: Add edifact-messages to cleanup_database.pl

This patch allows users to clear out old edifact_messages using the cleanup_database script. The number of days can either be set in the CLI or the default value of 365 can be used.

Test plan:
1) Ensure you have some EDI orders or even just some dummy messages in the edifact_messages table
2) Run perl misc/cronjobs/cleanup_database.pl --edifact-messages 100 --verbose (Change the number of days according to the data in your table)
3) The response should show a number of messages that would have been deleted
4) Run perl misc/cronjobs/cleanup_database.pl --edifact-messages 100 --verbose --confirm
5) The response should now show the same number of messages have been deleted
6) Check your edifact_messages table to confirm that the data has been deleted
Comment 2 Matt Blenkinsop 2023-02-15 15:55:33 UTC
Created attachment 146694 [details] [review]
Bug 30069: Add edifact-messages to cleanup_database.pl

This patch allows users to clear out old edifact_messages using the cleanup_database script. The number of days can either be set in the CLI or the default value of 365 can be used.

Test plan:
1) Ensure you have some EDI orders or even just some dummy messages in the edifact_messages table
2) Run perl misc/cronjobs/cleanup_database.pl --edifact-messages 100 --verbose (Change the number of days according to the data in your table)
3) The response should show a number of messages that would have been deleted
4) Run perl misc/cronjobs/cleanup_database.pl --edifact-messages 100 --verbose --confirm
5) The response should now show the same number of messages have been deleted
6) Check your edifact_messages table to confirm that the data has been deleted

Sponsored-by: PTFS Europe
Comment 3 Katrin Fischer 2023-02-20 21:30:57 UTC
Created attachment 146976 [details] [review]
Bug 30069: Add edifact-messages to cleanup_database.pl

This patch allows users to clear out old edifact_messages using the cleanup_database script. The number of days can either be set in the CLI or the default value of 365 can be used.

Test plan:
1) Ensure you have some EDI orders or even just some dummy messages in the edifact_messages table
2) Run perl misc/cronjobs/cleanup_database.pl --edifact-messages 100 --verbose (Change the number of days according to the data in your table)
3) The response should show a number of messages that would have been deleted
4) Run perl misc/cronjobs/cleanup_database.pl --edifact-messages 100 --verbose --confirm
5) The response should now show the same number of messages have been deleted
6) Check your edifact_messages table to confirm that the data has been deleted

Sponsored-by: PTFS Europe
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 4 Katrin Fischer 2023-02-20 21:31:00 UTC
Created attachment 146977 [details] [review]
Bug 30069: (QA follow-up) Fix capitalization for edifact

Changes a few occurrences of edifact in the output messages
to EDIFACT.

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 5 Katrin Fischer 2023-02-20 21:32:44 UTC
I have no experience with the EDIFACT process, so I am not sure what role the EDIFACT messages play. Could this cause any issues if the orders haven't been received yet?
Comment 6 Kyle M Hall (khall) 2023-02-21 11:59:38 UTC
(In reply to Katrin Fischer from comment #5)
> I have no experience with the EDIFACT process, so I am not sure what role
> the EDIFACT messages play. Could this cause any issues if the orders haven't
> been received yet?

Yes, that's a good thought. I think the query should but updated to not delete messages marked as 'new'. If automatic invoice importing is disabled, this feature should delete un-imported EDI invoices!
Comment 7 Matt Blenkinsop 2023-03-01 15:17:01 UTC
Created attachment 147564 [details] [review]
Bug 30069: (QA follow-up) Rebase and add filter

This patch now checks the status of messages and ignores any message
with a status of 'new'. It is also rebased to account for changes made
to cleanup_database.pl in bug 17350.

Test plan:
1) Ensure you have some EDI orders or even just some dummy messages in the edifact_messages table with a mixture of statuses including 'new'
2) Run perl misc/cronjobs/cleanup_database.pl --edifact-messages 100 --verbose (Change the number of days according to the data in your table)
3) The response should show a number of messages that would have been deleted
4) Run perl misc/cronjobs/cleanup_database.pl --edifact-messages 100 --verbose --confirm
5) The response should now show the same number of messages have been deleted
6) Check your edifact_messages table to confirm that the data has been deleted
7) Confirm that no messages marked 'new' have been deleted
Comment 8 Matt Blenkinsop 2023-03-01 15:18:28 UTC
Patch has been rebased and the requested check for 'new' messages has been added. Kyle I'm not too sure I understand what you are asking for in the second part of your comment about un-imported invoices, could you clarify further? Thanks
Comment 9 Kyle M Hall (khall) 2023-03-02 11:47:26 UTC
(In reply to Matt Blenkinsop from comment #8)
> Patch has been rebased and the requested check for 'new' messages has been
> added. Kyle I'm not too sure I understand what you are asking for in the
> second part of your comment about un-imported invoices, could you clarify
> further? Thanks

Invoices may not be processed and imported as soon as they are downloaded. It depends on the syspref  EdifactInvoiceImport
Comment 10 Kyle M Hall (khall) 2023-03-02 13:00:05 UTC
Created attachment 147606 [details] [review]
Bug 30069: Add edifact-messages to cleanup_database.pl

This patch allows users to clear out old edifact_messages using the cleanup_database script. The number of days can either be set in the CLI or the default value of 365 can be used.

Test plan:
1) Ensure you have some EDI orders or even just some dummy messages in the edifact_messages table
2) Run perl misc/cronjobs/cleanup_database.pl --edifact-messages 100 --verbose (Change the number of days according to the data in your table)
3) The response should show a number of messages that would have been deleted
4) Run perl misc/cronjobs/cleanup_database.pl --edifact-messages 100 --verbose --confirm
5) The response should now show the same number of messages have been deleted
6) Check your edifact_messages table to confirm that the data has been deleted

Sponsored-by: PTFS Europe
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 11 Kyle M Hall (khall) 2023-03-02 13:00:13 UTC
Created attachment 147607 [details] [review]
Bug 30069: (QA follow-up) Fix capitalization for edifact

Changes a few occurrences of edifact in the output messages
to EDIFACT.

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 12 Kyle M Hall (khall) 2023-03-02 13:00:15 UTC
Created attachment 147608 [details] [review]
Bug 30069: (QA follow-up) Rebase and add filter

This patch now checks the status of messages and ignores any message
with a status of 'new'. It is also rebased to account for changes made
to cleanup_database.pl in bug 17350.

Test plan:
1) Ensure you have some EDI orders or even just some dummy messages in the edifact_messages table with a mixture of statuses including 'new'
2) Run perl misc/cronjobs/cleanup_database.pl --edifact-messages 100 --verbose (Change the number of days according to the data in your table)
3) The response should show a number of messages that would have been deleted
4) Run perl misc/cronjobs/cleanup_database.pl --edifact-messages 100 --verbose --confirm
5) The response should now show the same number of messages have been deleted
6) Check your edifact_messages table to confirm that the data has been deleted
7) Confirm that no messages marked 'new' have been deleted

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 13 Kyle M Hall (khall) 2023-03-02 13:00:18 UTC
Created attachment 147609 [details] [review]
Bug 30069: (QA follow-up) Use pure DBIC for select and delete with single query

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 14 Tomás Cohen Arazi (tcohen) 2023-03-02 14:59:29 UTC
Pushed to master for 23.05.

Nice work everyone, thanks!
Comment 15 Matt Blenkinsop 2023-03-03 13:35:09 UTC
Not backporting to 22.11.x - enhancement
Comment 16 Aude Charillon 2024-08-28 09:37:47 UTC
Koha manual updated.