Bug 30824

Summary: Improve performance of BatchCommitItems
Product: Koha Reporter: Nick Clemens <nick>
Component: Architecture, internals, and plumbingAssignee: Nick Clemens <nick>
Status: CLOSED FIXED QA Contact: Joonas Kylmälä <joonas.kylmala>
Severity: normal    
Priority: P5 - low CC: jonathan.druart, joonas.kylmala, martin.renvoize, tomascohen
Version: master   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30822
Change sponsored?: --- Patch complexity: Small patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
22.11.00
Bug Depends on: 30813    
Bug Blocks:    
Attachments: Bug 30824: Improve performance of BatchCommitItems
Bug 30824: Improve performance of BatchCommitItems
Bug 30824: (follow-up) POD
Bug 30824: Improve performance of BatchCommitItems
Bug 30824: (follow-up) POD

Description Nick Clemens 2022-05-20 14:29:51 UTC

    
Comment 1 Nick Clemens 2022-05-20 14:36:04 UTC
Created attachment 135247 [details] [review]
Bug 30824: Improve performance of BatchCommitItems

This makes two simple changes:
- Limit TransformMarcToKoha to the fields we need
- Pass forward the biblioitemnumber when adding items to a new biblionumber

Profiling with NYTProf I saved ~8-9 seconds importing around 400 bibs/1000 items

Reducing calls in item store to use a passed biblionumber was the largest gain.

To test:
1 - Import some records and items
2 - Verify values etc., revert
3 - Apply patch
4 - Import again
5 - Verify values etc. same as before
Comment 2 Martin Renvoize 2022-05-24 14:29:46 UTC
Created attachment 135307 [details] [review]
Bug 30824: Improve performance of BatchCommitItems

This makes two simple changes:
- Limit TransformMarcToKoha to the fields we need
- Pass forward the biblioitemnumber when adding items to a new biblionumber

Profiling with NYTProf I saved ~8-9 seconds importing around 400 bibs/1000 items

Reducing calls in item store to use a passed biblionumber was the largest gain.

To test:
1 - Import some records and items
2 - Verify values etc., revert
3 - Apply patch
4 - Import again
5 - Verify values etc. same as before

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 3 Martin Renvoize 2022-05-24 14:29:50 UTC
Created attachment 135308 [details] [review]
Bug 30824: (follow-up) POD
Comment 4 Martin Renvoize 2022-05-24 14:30:15 UTC
Strong performance improvement, signing off.
Comment 5 Joonas Kylmälä 2022-05-26 11:40:30 UTC
Created attachment 135369 [details] [review]
Bug 30824: Improve performance of BatchCommitItems

This makes two simple changes:
- Limit TransformMarcToKoha to the fields we need
- Pass forward the biblioitemnumber when adding items to a new biblionumber

Profiling with NYTProf I saved ~8-9 seconds importing around 400 bibs/1000 items

Reducing calls in item store to use a passed biblionumber was the largest gain.

To test:
1 - Import some records and items
2 - Verify values etc., revert
3 - Apply patch
4 - Import again
5 - Verify values etc. same as before

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Comment 6 Joonas Kylmälä 2022-05-26 11:40:35 UTC
Created attachment 135370 [details] [review]
Bug 30824: (follow-up) POD

Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Comment 7 Joonas Kylmälä 2022-05-26 11:42:46 UTC
Makes the code harder to understand but I'll allow it due to the perf improvement. Passing QA.
Comment 8 Tomás Cohen Arazi 2022-06-06 16:55:57 UTC
How does the dependency work, Nick?
Comment 9 Nick Clemens 2022-06-06 17:20:47 UTC
(In reply to Tomás Cohen Arazi from comment #8)
> How does the dependency work, Nick?

It comes from the updated call to TransformMarcToKoha - we could simply use 'items' in the limit_table param to remove it
Comment 10 Tomás Cohen Arazi 2022-06-06 17:26:36 UTC
Blocked by tree.
Comment 11 Martin Renvoize 2022-06-10 10:57:58 UTC
Blocking patch has been pushed now, setting back to PQA
Comment 12 Tomás Cohen Arazi 2022-06-13 23:18:23 UTC
Pushed to master for 22.11.

Nice work everyone, thanks!