Bug 35165 - Batch item deletions are run in a single transaction and cause locking issues
Summary: Batch item deletions are run in a single transaction and cause locking issues
Status: Failed QA
Alias: None
Product: Koha
Classification: Unclassified
Component: Tools (show other bugs)
Version: Main
Hardware: All All
: P5 - low major
Assignee: Nick Clemens (kidclamp)
QA Contact: Marcel de Rooy
URL:
Keywords:
: 26077 (view as bug list)
Depends on:
Blocks:
 
Reported: 2023-10-26 13:59 UTC by Nick Clemens (kidclamp)
Modified: 2024-01-30 12:44 UTC (History)
11 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments
Bug 35165: Do not delete items in a transaction (1.94 KB, patch)
2024-01-19 19:16 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 35165: Tidy (6.93 KB, patch)
2024-01-19 19:16 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 35165: Do not delete items in a transaction (2.00 KB, patch)
2024-01-24 20:29 UTC, ByWater Sandboxes
Details | Diff | Splinter Review
Bug 35165: Tidy (6.99 KB, patch)
2024-01-24 20:29 UTC, ByWater Sandboxes
Details | Diff | Splinter Review
Bug 35165: Tidy (6.99 KB, patch)
2024-01-24 20:36 UTC, ByWater Sandboxes
Details | Diff | Splinter Review
Bug 35165: Do not delete items in a transaction (7.71 KB, patch)
2024-01-26 07:52 UTC, Marcel de Rooy
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Nick Clemens (kidclamp) 2023-10-26 13:59:44 UTC
We recently saw a site perform a batch deletion of ~18k items (and subsequently 2k bibs)

While running the jobs we suddenly started seeing all sorts of actions fail, most notably placing reserves, and errors in the logs like:
DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::mysql::st execute failed: Lock wait timeout exceeded; try restarting transaction at /usr/share/koha/lib/Koha/Object.pm line 170

I believe part of the problem is that the entire batch deletion is a single transaction.

While we should make parts of this atomic, I don't think the entire job should succeed or fail on an error. I think we should commit each item, then each bib, then continue.
Comment 1 Nick Clemens (kidclamp) 2023-11-03 11:32:48 UTC
*** Bug 26077 has been marked as a duplicate of this bug. ***
Comment 2 Nick Clemens (kidclamp) 2024-01-19 19:16:48 UTC
Created attachment 161219 [details] [review]
Bug 35165: Do not delete items in a transaction

We are testing the deletion with safe_delete before removing, beyond that I think the intent
was to delete all of the items, deleting some is better than not deleting any.

To test:
1 - Apply patch, restart all
2 - Submit a batch deletion
3 - Confirm items are deleted
Comment 3 Nick Clemens (kidclamp) 2024-01-19 19:16:50 UTC
Created attachment 161220 [details] [review]
Bug 35165: Tidy
Comment 4 Nick Clemens (kidclamp) 2024-01-19 19:18:28 UTC
(In reply to Nick Clemens from comment #0)
> While we should make parts of this atomic, I don't think the entire job
> should succeed or fail on an error. I think we should commit each item, then
> each bib, then continue.

Each item and bib are singular actions - I don't actually think we need any transactions at this level at all.
Comment 5 Loïc 2024-01-22 09:39:42 UTC
In "Details of job" after the deletion we have "Progress: 0/4" instead of "Progress: 4/4" without patch.

In the same page, in "Report" we have "No items deleted." instead of "4 item(s) deleted." without patch.
Comment 6 Loïc 2024-01-22 09:45:33 UTC
The bug appears just if we select "Delete records if no items remain."
Comment 7 Nick Clemens (kidclamp) 2024-01-23 19:49:46 UTC
(In reply to Loïc from comment #5)
> In "Details of job" after the deletion we have "Progress: 0/4" instead of
> "Progress: 4/4" without patch.
> 
> In the same page, in "Report" we have "No items deleted." instead of "4
> item(s) deleted." without patch.

(In reply to Loïc from comment #6)
> The bug appears just if we select "Delete records if no items remain."

I do not recreate this - did you restart all after applying the patch?
Comment 8 Jonathan Druart 2024-01-24 07:43:38 UTC
> While we should make parts of this atomic, I don't think the entire job should succeed or fail on an error.

I think it should. We should discuss that with other people. If something unexpected happened we should not commit.
Comment 9 Nick Clemens (kidclamp) 2024-01-24 13:02:54 UTC
(In reply to Jonathan Druart from comment #8)
> > While we should make parts of this atomic, I don't think the entire job should succeed or fail on an error.
> 
> I think it should. We should discuss that with other people. If something
> unexpected happened we should not commit.

The current situation places an unknown limit on the number that can be deleted - too many and you lock your system, can't circulate, can't catalog - this is very bad
Comment 10 Andrew Fuerste-Henry 2024-01-24 20:18:03 UTC
(In reply to Jonathan Druart from comment #8)
 
> I think it should. 

Can you elaborate? Nick presents a clear impact on the end-user and what appears to be a reasonable solution.
Comment 11 ByWater Sandboxes 2024-01-24 20:29:53 UTC
Created attachment 161368 [details] [review]
Bug 35165: Do not delete items in a transaction

We are testing the deletion with safe_delete before removing, beyond that I think the intent
was to delete all of the items, deleting some is better than not deleting any.

To test:
1 - Apply patch, restart all
2 - Submit a batch deletion
3 - Confirm items are deleted

Signed-off-by: Andrew Fuerste-Henry <andrewfh@dubcolib.org>
Comment 12 ByWater Sandboxes 2024-01-24 20:29:55 UTC
Created attachment 161369 [details] [review]
Bug 35165: Tidy

Signed-off-by: Andrew Fuerste-Henry <andrewfh@dubcolib.org>
Comment 13 Andrew Fuerste-Henry 2024-01-24 20:33:33 UTC
(In reply to Jonathan Druart from comment #8)
> If something
> unexpected happened we should not commit.

There's no expectation from the user that the entire batch of items will or will not delete as a group. So long as the user can reliably see which records were successfully deleted and which were not, the success of the job as a whole should not require every item succeed.
Comment 14 ByWater Sandboxes 2024-01-24 20:36:41 UTC
Created attachment 161370 [details] [review]
Bug 35165: Tidy

Signed-off-by: Andrew Fuerste-Henry <andrewfh@dubcolib.org>
Comment 15 Jonathan Druart 2024-01-25 07:55:42 UTC
This is how I implemented the background job at the beginning. If we decide to change how it should behave I am fine with that, but then we should have the same behaviour everywhere and make it clear (for end-users and devs).
Comment 16 Jonathan Druart 2024-01-25 07:57:41 UTC
The problem is mainly with the "retry". Say you have a "delete bib from 1 to 10". The first job deletes 1 to 5 but fails for 6 to 10. If you retry later after you fixed the deletion problem for 6-10, the second job (retry) will report failures for deletion 1 to 5.
Comment 17 Marcel de Rooy 2024-01-26 07:52:50 UTC
Created attachment 161480 [details] [review]
Bug 35165: Do not delete items in a transaction

We are testing the deletion with safe_delete before removing, beyond that I think the intent
was to delete all of the items, deleting some is better than not deleting any.

To test:
1 - Apply patch, restart all
2 - Submit a batch deletion
3 - Confirm items are deleted

Signed-off-by: Andrew Fuerste-Henry <andrewfh@dubcolib.org>
Comment 18 Marcel de Rooy 2024-01-26 07:53:31 UTC
We should avoid separate tidy patches.
Comment 19 Marcel de Rooy 2024-01-26 07:53:55 UTC
Apart from discussion, I dont see unit tests.
Comment 20 Nick Clemens (kidclamp) 2024-01-30 12:34:50 UTC
(In reply to Marcel de Rooy from comment #18)
> We should avoid separate tidy patches.

When we make a change that reindents a whole section, keeping the tidy patch separate makes rebasing far easier. Why should we avoid them?
Comment 21 Marcel de Rooy 2024-01-30 12:44:59 UTC
(In reply to Nick Clemens from comment #20)
> (In reply to Marcel de Rooy from comment #18)
> > We should avoid separate tidy patches.
> 
> When we make a change that reindents a whole section, keeping the tidy patch
> separate makes rebasing far easier. Why should we avoid them?

General rules always have exceptions :)