Bug 30824 - Improve performance of BatchCommitItems
Summary: Improve performance of BatchCommitItems
Status: Pushed to master
Alias: None
Product: Koha
Classification: Unclassified
Component: Architecture, internals, and plumbing (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: Nick Clemens
QA Contact: Joonas Kylmälä
URL:
Keywords:
Depends on: 30813
Blocks:
  Show dependency treegraph
 
Reported: 2022-05-20 14:29 UTC by Nick Clemens
Modified: 2022-06-13 23:18 UTC (History)
4 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
22.11.00


Attachments
Bug 30824: Improve performance of BatchCommitItems (4.61 KB, patch)
2022-05-20 14:36 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 30824: Improve performance of BatchCommitItems (4.68 KB, patch)
2022-05-24 14:29 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 30824: (follow-up) POD (1.12 KB, patch)
2022-05-24 14:29 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 30824: Improve performance of BatchCommitItems (4.82 KB, patch)
2022-05-26 11:40 UTC, Joonas Kylmälä
Details | Diff | Splinter Review
Bug 30824: (follow-up) POD (1.27 KB, patch)
2022-05-26 11:40 UTC, Joonas Kylmälä
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Nick Clemens 2022-05-20 14:29:51 UTC

    
Comment 1 Nick Clemens 2022-05-20 14:36:04 UTC
Created attachment 135247 [details] [review]
Bug 30824: Improve performance of BatchCommitItems

This makes two simple changes:
- Limit TransformMarcToKoha to the fields we need
- Pass forward the biblioitemnumber when adding items to a new biblionumber

Profiling with NYTProf I saved ~8-9 seconds importing around 400 bibs/1000 items

Reducing calls in item store to use a passed biblionumber was the largest gain.

To test:
1 - Import some records and items
2 - Verify values etc., revert
3 - Apply patch
4 - Import again
5 - Verify values etc. same as before
Comment 2 Martin Renvoize 2022-05-24 14:29:46 UTC
Created attachment 135307 [details] [review]
Bug 30824: Improve performance of BatchCommitItems

This makes two simple changes:
- Limit TransformMarcToKoha to the fields we need
- Pass forward the biblioitemnumber when adding items to a new biblionumber

Profiling with NYTProf I saved ~8-9 seconds importing around 400 bibs/1000 items

Reducing calls in item store to use a passed biblionumber was the largest gain.

To test:
1 - Import some records and items
2 - Verify values etc., revert
3 - Apply patch
4 - Import again
5 - Verify values etc. same as before

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 3 Martin Renvoize 2022-05-24 14:29:50 UTC
Created attachment 135308 [details] [review]
Bug 30824: (follow-up) POD
Comment 4 Martin Renvoize 2022-05-24 14:30:15 UTC
Strong performance improvement, signing off.
Comment 5 Joonas Kylmälä 2022-05-26 11:40:30 UTC
Created attachment 135369 [details] [review]
Bug 30824: Improve performance of BatchCommitItems

This makes two simple changes:
- Limit TransformMarcToKoha to the fields we need
- Pass forward the biblioitemnumber when adding items to a new biblionumber

Profiling with NYTProf I saved ~8-9 seconds importing around 400 bibs/1000 items

Reducing calls in item store to use a passed biblionumber was the largest gain.

To test:
1 - Import some records and items
2 - Verify values etc., revert
3 - Apply patch
4 - Import again
5 - Verify values etc. same as before

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Comment 6 Joonas Kylmälä 2022-05-26 11:40:35 UTC
Created attachment 135370 [details] [review]
Bug 30824: (follow-up) POD

Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Comment 7 Joonas Kylmälä 2022-05-26 11:42:46 UTC
Makes the code harder to understand but I'll allow it due to the perf improvement. Passing QA.
Comment 8 Tomás Cohen Arazi 2022-06-06 16:55:57 UTC
How does the dependency work, Nick?
Comment 9 Nick Clemens 2022-06-06 17:20:47 UTC
(In reply to Tomás Cohen Arazi from comment #8)
> How does the dependency work, Nick?

It comes from the updated call to TransformMarcToKoha - we could simply use 'items' in the limit_table param to remove it
Comment 10 Tomás Cohen Arazi 2022-06-06 17:26:36 UTC
Blocked by tree.
Comment 11 Martin Renvoize 2022-06-10 10:57:58 UTC
Blocking patch has been pushed now, setting back to PQA
Comment 12 Tomás Cohen Arazi 2022-06-13 23:18:23 UTC
Pushed to master for 22.11.

Nice work everyone, thanks!