Bug 8685 - Inventory tool times out with large barcode files
Summary: Inventory tool times out with large barcode files
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Tools (show other bugs)
Version: unspecified
Hardware: All All
: P5 - low normal with 10 votes (vote)
Assignee: Galen Charlton
QA Contact:
URL:
Keywords:
Depends on: 1535
Blocks:
  Show dependency treegraph
 
Reported: 2012-08-24 09:00 UTC by Sophie MEYNIEUX
Modified: 2020-01-06 20:15 UTC (History)
13 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Sophie MEYNIEUX 2012-08-24 09:00:43 UTC
Inventory tool fails with a timeout if the CB file to process is too large.
In fact it doesn't really fails, CB are properly processed, but the user can't see the result because of the timeout occuring before the end of processing.

One solution could be to implement background job like in MARC import (BZ1531)

Another one could be to add scheduler to plan inventory process like in reports
Comment 1 Paul Poulain 2012-08-28 10:24:21 UTC
(CB = code barre = barcode file)
Comment 2 Katrin Fischer 2014-07-12 12:31:35 UTC
Could this be related to bug 12025?
Comment 3 Jonathan Druart 2015-04-01 14:30:09 UTC
Sophie, do you know if this is still valid for recent version of Koha?
Comment 4 Mark Tompsett 2015-04-15 05:43:33 UTC
(In reply to Sophie MEYNIEUX from comment #0)
> Inventory tool fails with a timeout if the CB file to process is too large.
> In fact it doesn't really fails, CB are properly processed, but the user
> can't see the result because of the timeout occuring before the end of
> processing.
> 
> One solution could be to implement background job like in MARC import
> (BZ1531)

I think this would be a decent idea.


> Another one could be to add scheduler to plan inventory process like in
> reports

I like the background job idea better, because if you are there are the inventory/stocktaking tool page, you want that done then and there. A pretty progress back gives nice feedback too.


(In reply to Jonathan Druart from comment #3)
> Sophie, do you know if this is still valid for recent version of Koha?

My testing for bug 10625 for an unfiltered report ran for just over 1 minute for just a couple of barcodes. I can easily imagine that a large barcode file would generate a timeout in terms of the web page not displaying before the script completed.
Comment 5 Mark Tompsett 2015-05-04 18:26:29 UTC
Katrin suggested this could be a duplicate for bug 12025, but perhaps I also resolved it with bug 14057. Perhaps this should be resolved/duplicate closed?
Comment 6 Nicole C. Engard 2015-06-12 17:48:24 UTC
Has anyone confirmed that this is still an issue/resolved?
Comment 7 Josef Moravec 2017-01-11 19:49:10 UTC
It's still an issue, also after bug 14057. 

I did just few thousands inventory, and I faced time out (version 16.05.04).

I like the background job idea...
Comment 8 Danielle Elder 2017-06-23 00:07:22 UTC
Still happening and affecting libraries wanting to do inventories.
Comment 9 Danielle Elder 2017-06-23 00:07:31 UTC
Still happening and affecting libraries wanting to do inventories.
Comment 10 Mark Tompsett 2017-06-23 03:18:02 UTC
(In reply to Josef Moravec from comment #7)
> I did just few thousands inventory, and I faced time out (version 16.05.04).

How many barcodes? Exact number is useful for attempting to duplicate failure. :)
Comment 11 Josef Moravec 2017-06-23 07:30:57 UTC
(In reply to M. Tompsett from comment #10)
> (In reply to Josef Moravec from comment #7)
> > I did just few thousands inventory, and I faced time out (version 16.05.04).
> 
> How many barcodes? Exact number is useful for attempting to duplicate
> failure. :)

Exactly it was 4109 barcodes at once ;)
Comment 12 Michal Denar 2017-06-23 07:45:41 UTC
It is not depend on number of codes, but time that Koha consume to process them. Think that this is problem with websere timeout time. If it is set on 600 s (default), problem can be relevant for about 500 - 1000 barcodes. So, we need solve it by universal way = make Inventory tool improvements that make processing of barcodes on background, similar like import staging. Webserver will not wait for it and timeouts will gone. "Small" change can make Inventory tool really usefull.
Comment 13 Danielle Elder 2017-06-23 19:45:15 UTC
The library I have am working with has to do 2,000 at time, any more and it times out.
Comment 14 Mason James 2018-03-28 05:16:26 UTC
a temporary workaround for this issue is to increase the apache TimeOut variable to something large.. like 900 (or more)

 https://httpd.apache.org/docs/current/mod/core.html#timeout

900 seconds = 15 minutes, apache's default is 60

you can add the 'Timeout 900' line to your apache.conf (global)
..but you probably just want to add the line inside your Koha intranet vhost(s)


we've been running this fix on prod without any problems
Comment 15 emjhet 2018-05-30 22:14:36 UTC
This is a huge issue for our libraries. Currently the only way to limit inventory is by library and shelving location. At some of our libraries, we can not limit by call number because some collections do not have call numbers. By not providing any other ways to limit an inventory, we have no choice but to process collections of over 1000 items. I have had to delay our inventories because of this, and the scanning that I did do was a wasted few days because the data could not be processed.
Comment 16 abernaud 2018-06-07 08:54:01 UTC
Hello,
this is a problem for our library too (we're on Koha 16.05.11).
We'll use the workaround mentionned in Comment#14, but our IT department is not very happy with this... However, this seems to let us upload relatively large files of barcodes (~3000). But the upload is still very long...
We'd like to be able to upload 30 files of 3000 barcodes at the end of each half-day of inventory. This upload is not supposed to take us a full hour (but it will probably be the case...)

We're ready to test some improvements proposed so solve this problem !
Thanks
Anne-Claire (Rennes 1, France)
Comment 17 Jonathan Druart 2018-06-07 14:10:44 UTC
(In reply to abernaud from comment #16)
> Hello,
> this is a problem for our library too (we're on Koha 16.05.11).
> We'll use the workaround mentionned in Comment#14, but our IT department is
> not very happy with this... However, this seems to let us upload relatively
> large files of barcodes (~3000). But the upload is still very long...
> We'd like to be able to upload 30 files of 3000 barcodes at the end of each
> half-day of inventory. This upload is not supposed to take us a full hour
> (but it will probably be the case...)
> 
> We're ready to test some improvements proposed so solve this problem !
> Thanks
> Anne-Claire (Rennes 1, France)

Hi Anne-Claire,
FYI there is a performance patch for the inventory (bug 19023) that is not part of 16.05.x
I can take a look but I will need numbers with the last Koha version under a Plack setup
Comment 18 Jonathan Druart 2018-06-07 14:12:30 UTC
(In reply to Jonathan Druart from comment #17)
> FYI there is a performance patch for the inventory (bug 19023) that is not
> part of 16.05.x

Hum, forget that, bug 19023 fixed performance issues caused by bug 17249, which is not in 16.05.x
Comment 19 abernaud 2018-07-06 07:40:37 UTC
OK, thanks for trying to help, Jonathan !
Comment 20 emjhet 2018-07-11 18:23:29 UTC
(In reply to Mike from comment #12)
> It is not depend on number of codes, but time that Koha consume to process
> them. Think that this is problem with websere timeout time. If it is set on
> 600 s (default), problem can be relevant for about 500 - 1000 barcodes. So,
> we need solve it by universal way = make Inventory tool improvements that
> make processing of barcodes on background, similar like import staging.
> Webserver will not wait for it and timeouts will gone. "Small" change can
> make Inventory tool really usefu(In reply to Mike from comment #12)
> It is not depend on number of codes, but time that Koha consume to process
> them. Think that this is problem with websere timeout time. If it is set on
> 600 s (default), problem can be relevant for about 500 - 1000 barcodes. So,
> we need solve it by universal way = make Inventory tool improvements that
> make processing of barcodes on background, similar like import staging.
> Webserver will not wait for it and timeouts will gone. "Small" change can
> make Inventory tool really usefull.

So is this still a problem in Koha 17.11.06? I don't want to take the time to scan thousands of items just to have it not work (again.) I believe I was using 17.05 the last time it did not work. What do we need to do to get the results from the inventory process? Not having this ability really limits the usefulness of Koha.
Comment 21 Katrin Fischer 2018-07-14 11:55:42 UTC
 
> So is this still a problem in Koha 17.11.06? I don't want to take the time
> to scan thousands of items just to have it not work (again.) I believe I was
> using 17.05 the last time it did not work. What do we need to do to get the
> results from the inventory process? Not having this ability really limits
> the usefulness of Koha

I think noone was able to retest this yet. If you have a test system available to you, you could create a report in order to get a long list of barcodes and test with that. That could help to verify if the bug still exists.
Comment 22 emjhet 2018-07-18 18:06:50 UTC
(In reply to Katrin Fischer from comment #21)
>  
> I think noone was able to retest this yet. If you have a test system
> available to you, you could create a report in order to get a long list of
> barcodes and test with that. That could help to verify if the bug still
> exists.

I am using Koha 17.11.06. I ran a test batch through the inventory module. It had 3747 items in it and it did not time out!

It may be because the results are coming back differently. Instead of getting results that list each barcode and the fact that the item's status was updated, only the "problem" items were listed (things that were checked out, missing, etc.) And it actually used the date that I told it to (which it did one time I ran an inventory and another time it did not.)

So this may not be a problem any longer. I do not have another library location that I can run a fake inventory on at this time to see if a larger file would run successfully. Please do note that we do not have the fastest internet here, so those with better connections may be able to run larger files.
Comment 23 Katrin Fischer 2018-07-18 18:37:42 UTC
Thx for testing and sharing your results! Glad that the situation seems to have improved!
Comment 24 Katrin Fischer 2019-04-28 12:21:25 UTC
Judging from latest comments this has been resolved. Please reopen with your version and number of barcodes used if necessary.