Bug 19722 - Add a MaxItemsToDisplayForBatchMod preference
Summary: Add a MaxItemsToDisplayForBatchMod preference
Status: RESOLVED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Tools (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Nick Clemens
QA Contact: Josef Moravec
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-11-30 15:54 UTC by Nick Clemens
Modified: 2019-05-30 12:24 UTC (History)
7 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Who signed the patch off:
Text to go in the release notes:
When batch editing large amounts of items, displaying all of the info could lead to a timeout while waiting for page load. This enhancement adds the ability to define a MaxItemsToDisplayForBatchMod system preference which will hide the list of individual items and allow libraries to edit larger batches without hitting a timeout.
Version(s) released in:


Attachments
Bug 19722 - Add MaxItemsToDisplayForBatchMod systempreference (7.34 KB, patch)
2017-11-30 23:29 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 19722 - Add MaxItemsToDisplayForBatchMod systempreference (7.49 KB, patch)
2018-04-21 10:07 UTC, Séverine Queune
Details | Diff | Splinter Review
Bug 19722: (follow-up) Adjust template parameters (2.44 KB, patch)
2018-07-16 17:44 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 19722 - Add MaxItemsToDisplayForBatchMod systempreference (8.14 KB, patch)
2019-03-26 20:23 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 19722: (follow-up) Adjust template parameters (3.10 KB, patch)
2019-03-26 20:23 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 19722: Don't show items at job completion if over pref (4.52 KB, patch)
2019-03-26 20:23 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 19722: Add MaxItemsToDisplayForBatchMod systempreference (8.20 KB, patch)
2019-03-27 15:49 UTC, Josef Moravec
Details | Diff | Splinter Review
Bug 19722: (follow-up) Adjust template parameters (3.16 KB, patch)
2019-03-27 15:50 UTC, Josef Moravec
Details | Diff | Splinter Review
Bug 19722: Don't show items at job completion if over pref (4.58 KB, patch)
2019-03-27 15:50 UTC, Josef Moravec
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Nick Clemens 2017-11-30 15:54:04 UTC
The tool is setup to run in the background for actual modifications, but displaying the items beforehand can take too long and cause timeouts.

We should add a syspref and use similar functionality from batch delete items where one can edit more than is viewed
Comment 1 Nick Clemens 2017-11-30 23:29:29 UTC
Created attachment 69426 [details] [review]
Bug 19722 - Add MaxItemsToDisplayForBatchMod systempreference

To test:
1 - Get a list of itemnumbers or barcodes
2 - Set MaxItemsToProcessForBatchMod to less than this number of items
3 - Attempt to batch edit, you arent allowed
4 - Set MaxItemsToProcessForBatchMod to some hugh number like 60000
5 - Try a list of as many items as possible 30000 was good for me (they
        should exist in your system)
6 - Try to batch edit, system will timeout on displaying these items
7 - Apply patch, update database
8 - Set MaxItemsToDisplayForBatchMod to 1000 (default)
9 - Try large file again, should not display items, but allow
modification, wait for this to finish
10 - SetMaxItemsToProcessForBatchMod to 1000 and try large file again
11 - Should be prohibited as before
Comment 2 jmbroust 2018-03-13 12:57:49 UTC
Point 9 and 10 seem to be the same action with two different behaviors waited. Am I missing something ? Could you have a look and explain, please ?
Comment 3 Nick Clemens 2018-03-13 14:28:05 UTC
(In reply to jmbroust from comment #2)
> Point 9 and 10 seem to be the same action with two different behaviors
> waited. Am I missing something ? Could you have a look and explain, please ?

Ah, they are testing two different preferences:
MaxItemsTo Display ForBatchMod
MaxItemsTo Process ForBatchMod

If process is higher than display you should be able to process as many as process and they just won't show

If process is lower and you try to edit more, you should be restricted.
Comment 4 abernaud 2018-03-15 11:03:09 UTC
Unable to test step 5 et 6 : not enough items in the sandbox

At step 9 : the "job progress" bar stays at 0% and I get an error message : "Failed to submit form : error".
Nothing happens after that.

Other steps are OK.
Comment 5 Nick Clemens 2018-03-30 13:12:45 UTC
(In reply to abernaud from comment #4)
> Unable to test step 5 et 6 : not enough items in the sandbox
> 
> At step 9 : the "job progress" bar stays at 0% and I get an error message :
> "Failed to submit form : error".
> Nothing happens after that.
> 
> Other steps are OK.

You can add 1000+ items to a record in the edit items screen to easily get a large number

Step 9 worked fine for me, can you check for JS errors and/or retest?
Comment 6 Bouzid Fergani 2018-04-20 20:21:13 UTC
Hi Nick,
Yes it work, but I have problem with MaxItemsToProcessForBatchMod, it's not respected.
For example, 
    - I use file of 21 barcode
    - MaxItemsToProcessForBatchMod = 15
The batch item modification open correctly, not as without this patch, but it open all items in the file, not 15 as specify in MaxItemsToProcessForBatchMod.
 
Thanks.
Bouzid Fergani
Comment 7 Séverine Queune 2018-04-21 10:07:57 UTC
Created attachment 74682 [details] [review]
Bug 19722 - Add MaxItemsToDisplayForBatchMod systempreference

To test:
1 - Get a list of itemnumbers or barcodes
2 - Set MaxItemsToProcessForBatchMod to less than this number of items
3 - Attempt to batch edit, you arent allowed
4 - Set MaxItemsToProcessForBatchMod to some hugh number like 60000
5 - Try a list of as many items as possible 30000 was good for me (they
        should exist in your system)
6 - Try to batch edit, system will timeout on displaying these items
7 - Apply patch, update database
8 - Set MaxItemsToDisplayForBatchMod to 1000 (default)
9 - Try large file again, should not display items, but allow
modification, wait for this to finish
10 - SetMaxItemsToProcessForBatchMod to 1000 and try large file again
11 - Should be prohibited as before

Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Comment 8 Séverine Queune 2018-04-21 10:15:54 UTC
(In reply to Nick Clemens from comment #5)
> (In reply to abernaud from comment #4)
> > 
> > At step 9 : the "job progress" bar stays at 0% and I get an error message :
> > "Failed to submit form : error".
> > Nothing happens after that.
> 
> Step 9 worked fine for me, can you check for JS errors and/or retest?

I had the same problem as abernaud when I tried to modify 1500 items with this parameters : allowed process = 2000 / display = 1000
I realize I was just impatient :) When I get this error, modification was very long and "Job process" not moving, but no more error when I tried again and waited to the end of modification, or when I tried with smaller quantity of items.

Seems ok for me, thank you Nick !
Comment 9 Jonathan Druart 2018-05-31 19:24:27 UTC
Batch item deletion is broken, please double check.
I always get "0 item(s) deleted."
Comment 10 Nick Clemens 2018-07-16 17:44:17 UTC
Created attachment 77020 [details] [review]
Bug 19722: (follow-up) Adjust template parameters

Before we passed 'too_many_items' this was renamed to
'too_many_items_display' but deletion template lacked changes

To test:
1 - Test a batch deletion by list and file
2 - Test with deleting biblios and without
3 - Results should be as expected
Comment 11 Jonathan Druart 2018-07-23 17:37:50 UTC
The items are displayed at the end of the process (and so can lead to a timeout).
Is it expected?
Comment 12 Nick Clemens 2018-08-06 17:27:54 UTC
You are correct, this should be updated to follow the system preference
Comment 13 Nick Clemens 2019-03-26 20:23:12 UTC
Created attachment 87033 [details] [review]
Bug 19722 - Add MaxItemsToDisplayForBatchMod systempreference

To test:
1 - Get a list of itemnumbers or barcodes
2 - Set MaxItemsToProcessForBatchMod to less than this number of items
3 - Attempt to batch edit, you arent allowed
4 - Set MaxItemsToProcessForBatchMod to some hugh number like 60000
5 - Try a list of as many items as possible 30000 was good for me (they
        should exist in your system)
6 - Try to batch edit, system will timeout on displaying these items
7 - Apply patch, update database
8 - Set MaxItemsToDisplayForBatchMod to 1000 (default)
9 - Try large file again, should not display items, but allow
modification, wait for this to finish
10 - SetMaxItemsToProcessForBatchMod to 1000 and try large file again
11 - Should be prohibited as before

Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Comment 14 Nick Clemens 2019-03-26 20:23:15 UTC
Created attachment 87034 [details] [review]
Bug 19722: (follow-up) Adjust template parameters

Before we passed 'too_many_items' this was renamed to
'too_many_items_display' but deletion template lacked changes

To test:
1 - Test a batch deletion by list and file
2 - Test with deleting biblios and without
3 - Results should be as expected
Comment 15 Nick Clemens 2019-03-26 20:23:18 UTC
Created attachment 87035 [details] [review]
Bug 19722: Don't show items at job completion if over pref

To test:
1 - Set MaxItemsToDisplayForBatchMod to some number
2 - Modify less items than that
3 - Note they show
    a - Note if over 1000 they show in a simple display
4 - Modify more items
5 - Note they do not show and you see a note
Comment 16 Josef Moravec 2019-03-27 15:49:59 UTC
Created attachment 87077 [details] [review]
Bug 19722: Add MaxItemsToDisplayForBatchMod systempreference

To test:
1 - Get a list of itemnumbers or barcodes
2 - Set MaxItemsToProcessForBatchMod to less than this number of items
3 - Attempt to batch edit, you arent allowed
4 - Set MaxItemsToProcessForBatchMod to some hugh number like 60000
5 - Try a list of as many items as possible 30000 was good for me (they
        should exist in your system)
6 - Try to batch edit, system will timeout on displaying these items
7 - Apply patch, update database
8 - Set MaxItemsToDisplayForBatchMod to 1000 (default)
9 - Try large file again, should not display items, but allow
modification, wait for this to finish
10 - SetMaxItemsToProcessForBatchMod to 1000 and try large file again
11 - Should be prohibited as before

Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Comment 17 Josef Moravec 2019-03-27 15:50:03 UTC
Created attachment 87078 [details] [review]
Bug 19722: (follow-up) Adjust template parameters

Before we passed 'too_many_items' this was renamed to
'too_many_items_display' but deletion template lacked changes

To test:
1 - Test a batch deletion by list and file
2 - Test with deleting biblios and without
3 - Results should be as expected

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Comment 18 Josef Moravec 2019-03-27 15:50:07 UTC
Created attachment 87079 [details] [review]
Bug 19722: Don't show items at job completion if over pref

To test:
1 - Set MaxItemsToDisplayForBatchMod to some number
2 - Modify less items than that
3 - Note they show
    a - Note if over 1000 they show in a simple display
4 - Modify more items
5 - Note they do not show and you see a note

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Comment 19 Nick Clemens 2019-03-28 18:19:07 UTC
Awesome work all!

Pushed to master for 19.05
Comment 20 Martin Renvoize 2019-04-01 13:59:46 UTC
Enhancement, will not be backported to 18.11.x series.