Bug 21006

Summary: GetMarcBiblio is slow for records with many items
Product: Koha Reporter: Nick Clemens <nick>
Component: Architecture, internals, and plumbingAssignee: Nick Clemens <nick>
Status: RESOLVED DUPLICATE QA Contact: Testopia <testopia>
Severity: enhancement    
Priority: P5 - low CC: ere.maijala, fridolin.somers, jonathan.druart, kyle, m.de.rooy, tomascohen
Version: Main   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20664
Change sponsored?: --- Patch complexity: Small patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Attachments: Bug 21006: Add GetMarcItems method and refactor GetMarcBiblio
Bug 21006: Add GetMarcItems method and refactor GetMarcBiblio
Bug 21006: Add unit tests for GetMarcItem
Bug 21006: Add GetMarcItems method and refactor GetMarcBiblio
Bug 21006: Unit tests for GetMarcItems

Description Nick Clemens 2018-06-27 02:58:49 UTC
Some libraries have serials that have large numbers of items attached to the record, this can create problems when exporting/harvesting the records.

One of the problems seems to be GetMarcBiblio which makes several db requests per item, many of these can be removed by adding a GetMarcItems routine to fetch several items rather than iterating over each biblionumber
Comment 1 Nick Clemens 2018-06-27 02:59:53 UTC
Created attachment 76475 [details] [review]
Bug 21006: Add GetMarcItems method and refactor GetMarcBiblio
Comment 2 Jonathan Druart 2018-06-27 12:54:09 UTC
Could you provide a patch with only the relevant changes (i.e. without indentation changes)?
Comment 3 Nick Clemens 2018-06-27 13:15:40 UTC
Created attachment 76490 [details] [review]
Bug 21006: Add GetMarcItems method and refactor GetMarcBiblio
Comment 4 Nick Clemens 2018-06-27 14:59:57 UTC
just to add, testing with a bib with 50,000 items I saw the time for a call to GetMarcBiblio (with embed_items) go from ~135 seconds to ~18 seconds
Comment 5 Jonathan Druart 2018-06-27 16:04:37 UTC
After a quick look at the code, to test later:
If item type at biblio level (item-level_itypes=0) then 952$y will not contain biblioitem.itemtype.

However I am not sure this make sense...
Comment 6 Nick Clemens 2018-07-04 01:27:52 UTC
Created attachment 76659 [details] [review]
Bug 21006: Add unit tests for GetMarcItem

To test:
prove -v t/db_dependent/Items.t
Comment 7 Nick Clemens 2018-07-04 01:27:55 UTC
Created attachment 76660 [details] [review]
Bug 21006: Add GetMarcItems method and refactor GetMarcBiblio
Comment 8 Nick Clemens 2018-07-04 01:27:59 UTC
Created attachment 76661 [details] [review]
Bug 21006: Unit tests for GetMarcItems

To test:
prove -v t/db_dependent/Items.t
Comment 9 Ere Maijala 2018-09-15 18:07:49 UTC
While this makes records with a large number of items faster, it slows down retrieval of records with only a few items. An alternative implementation in bug 20664.
Comment 10 Nick Clemens 2018-09-20 19:06:56 UTC
Ere demonstrated the issue for me - at least my tests were useful - closing this in favor of his
Comment 11 Nick Clemens 2018-09-20 19:07:23 UTC

*** This bug has been marked as a duplicate of bug 20664 ***