Bug 21006 - GetMarcBiblio is slow for records with many items
Summary: GetMarcBiblio is slow for records with many items
Status: RESOLVED DUPLICATE of bug 20664
Alias: None
Product: Koha
Classification: Unclassified
Component: Architecture, internals, and plumbing (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Nick Clemens
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-06-27 02:58 UTC by Nick Clemens
Modified: 2018-09-20 19:07 UTC (History)
6 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 21006: Add GetMarcItems method and refactor GetMarcBiblio (16.80 KB, patch)
2018-06-27 02:59 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 21006: Add GetMarcItems method and refactor GetMarcBiblio (3.04 KB, patch)
2018-06-27 13:15 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 21006: Add unit tests for GetMarcItem (2.86 KB, patch)
2018-07-04 01:27 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 21006: Add GetMarcItems method and refactor GetMarcBiblio (3.67 KB, patch)
2018-07-04 01:27 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 21006: Unit tests for GetMarcItems (5.31 KB, patch)
2018-07-04 01:27 UTC, Nick Clemens
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Nick Clemens 2018-06-27 02:58:49 UTC
Some libraries have serials that have large numbers of items attached to the record, this can create problems when exporting/harvesting the records.

One of the problems seems to be GetMarcBiblio which makes several db requests per item, many of these can be removed by adding a GetMarcItems routine to fetch several items rather than iterating over each biblionumber
Comment 1 Nick Clemens 2018-06-27 02:59:53 UTC
Created attachment 76475 [details] [review]
Bug 21006: Add GetMarcItems method and refactor GetMarcBiblio
Comment 2 Jonathan Druart 2018-06-27 12:54:09 UTC
Could you provide a patch with only the relevant changes (i.e. without indentation changes)?
Comment 3 Nick Clemens 2018-06-27 13:15:40 UTC
Created attachment 76490 [details] [review]
Bug 21006: Add GetMarcItems method and refactor GetMarcBiblio
Comment 4 Nick Clemens 2018-06-27 14:59:57 UTC
just to add, testing with a bib with 50,000 items I saw the time for a call to GetMarcBiblio (with embed_items) go from ~135 seconds to ~18 seconds
Comment 5 Jonathan Druart 2018-06-27 16:04:37 UTC
After a quick look at the code, to test later:
If item type at biblio level (item-level_itypes=0) then 952$y will not contain biblioitem.itemtype.

However I am not sure this make sense...
Comment 6 Nick Clemens 2018-07-04 01:27:52 UTC
Created attachment 76659 [details] [review]
Bug 21006: Add unit tests for GetMarcItem

To test:
prove -v t/db_dependent/Items.t
Comment 7 Nick Clemens 2018-07-04 01:27:55 UTC
Created attachment 76660 [details] [review]
Bug 21006: Add GetMarcItems method and refactor GetMarcBiblio
Comment 8 Nick Clemens 2018-07-04 01:27:59 UTC
Created attachment 76661 [details] [review]
Bug 21006: Unit tests for GetMarcItems

To test:
prove -v t/db_dependent/Items.t
Comment 9 Ere Maijala 2018-09-15 18:07:49 UTC
While this makes records with a large number of items faster, it slows down retrieval of records with only a few items. An alternative implementation in bug 20664.
Comment 10 Nick Clemens 2018-09-20 19:06:56 UTC
Ere demonstrated the issue for me - at least my tests were useful - closing this in favor of his
Comment 11 Nick Clemens 2018-09-20 19:07:23 UTC

*** This bug has been marked as a duplicate of bug 20664 ***