Bug 26557 - Batch import fails when incoming records contain itemnumber
Summary: Batch import fails when incoming records contain itemnumber
Status: RESOLVED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Tools (show other bugs)
Version: master
Hardware: All All
: P5 - low major (vote)
Assignee: Jonathan Druart
QA Contact: Testopia
URL:
Keywords:
Depends on: 23463
Blocks: 26543
  Show dependency treegraph
 
Reported: 2020-09-28 19:59 UTC by Andrew Fuerste-Henry
Modified: 2020-11-24 08:58 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Text to go in the release notes:
Version(s) released in:
20.11.00, 20.05.06


Attachments
Bug 26557: (bug 23463 follow-up) Fix Batch import when incoming records contain itemnumber (3.07 KB, patch)
2020-11-19 14:20 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 26557: (bug 23463 follow-up) Fix Batch import when incoming records contain itemnumber (3.12 KB, patch)
2020-11-19 18:35 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review
Bug 26557: (bug 23463 follow-up) Fix Batch import when incoming records contain itemnumber (3.16 KB, patch)
2020-11-19 18:47 UTC, David Nind
Details | Diff | Splinter Review
Bug 26557: (bug 23463 follow-up) Fix Batch import when incoming records contain itemnumber (3.17 KB, patch)
2020-11-19 19:53 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review
Bug 26557: (bug 23463 follow-up) Fix Batch import when incoming records contain itemnumber (3.23 KB, patch)
2020-11-20 11:03 UTC, Martin Renvoize
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Andrew Fuerste-Henry 2020-09-28 19:59:49 UTC
To recreate:
1 - find an existing bib in your system with just one item
2 - export that bib with the item attached
3 - delete the barcode from your item in Koha
4 - stage your exported marc file for reimport, match on biblionumber, set it to Always Add Items
5 - confirm that the bib matches and the incoming 952 is parsed
6 - click "Import this batch into the catalog"
7 - the Import button grays out, the job progress bar appears, but the import never progresses

I have confirmed this happens in master and 20.05. On 19.11 -- and previous versions, as far as I recall -- Koha makes a new item with a new itemnumber.
Comment 1 Jonathan Druart 2020-11-19 14:10:09 UTC
Pretty sure this is caused by bug 23463.
Comment 2 Jonathan Druart 2020-11-19 14:20:35 UTC
Created attachment 113816 [details] [review]
Bug 26557: (bug 23463 follow-up) Fix Batch import when incoming records contain itemnumber

Prior to ug 23463 AddItemFromMarc where calling AddItem, that did not
take into account the itemnumber field.
Now that we are using Koha::Item, we need to remove the items.itemnumber
field from the MARC record

Test plan:
1 - find an existing bib in your system with just one item
2 - export that bib with the item attached
3 - delete the barcode from your item in Koha
4 - stage your exported marc file for reimport, match on biblionumber, set it to Always Add Items
5 - confirm that the bib matches and the incoming 952 is parsed
6 - click "Import this batch into the catalog"

=> Without this patch you get (in the logs, or hidden)
manage-marc-import.pl: DBD::mysql::st execute failed: Duplicate entry '23' for key 'PRIMARY' [for Statement "INSERT INTO `items` ( `barcode`, `biblioitemnumber`, `biblionumber`, `ccode`, `cn_sort`, `cn_source`, `damaged_on`, `dateaccessioned`, `datelastborrowed`, `datelastseen`, `holdingbranch`, `homebranch`, `itemcallnumber`, `itemlost_on`, `itemnumber`, `itype`, `location`, `more_subfields_xml`, `onloan`, `permanent_location`, `replacementpricedate`, `timestamp`, `withdrawn_on`) VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, current_timestamp, ? )" with ParamValues: 0="BC_23", 1=8, 2=8, 3="REF", 4='CN__23', 5=undef, 6=undef, 7="2014-09-04", 8=undef, 9="2014-09-04", 10="FPL", 11="FPL", 12="CN_23", 13=undef, 14="23", 15="BK", 16="GEN", 17=undef, 18=undef, 19="GEN", 20="2014-09-04", 21=undef] at /usr/share/perl5/DBIx/Class/Storage/DBI.pm line 1836.
manage-marc-import.pl: DBIx::Class::Storage::DBI::_dbh_execute(): Duplicate entry '23' for key 'PRIMARY' at /kohadevbox/koha/Koha/Object.pm line 169
manage-marc-import.pl: {UNKNOWN}: Transaction aborted: Duplicate ID. Rollback failed: DBIx::Class::Storage::txn_rollback(): Refusing to roll back without a started transaction at /kohadevbox/koha/tools/manage-marc-import.pl line 253 at /kohadevbox/koha/tools/manage-marc-import.pl line 253

=> With this patch applied, the new item must be added to the existing bibliographic record
Comment 3 Victor Grousset/tuxayo 2020-11-19 18:34:50 UTC
I have the error in sip.log o_o does that happen to someone else?

> ==> sip.log <==
> [Thu Nov 19 18:23:32 2020] manage-marc-import.pl: DBD::mysql::st execute failed: Duplicate entry '969' for key 'PRIMARY'

my setup: koha-testing-docker


> 6 - click "Import this batch into the catalog"

Also in my case, the import just stay frozen.


Nonetheless, it works with the patch :)
Comment 4 Victor Grousset/tuxayo 2020-11-19 18:35:53 UTC
Created attachment 113844 [details] [review]
Bug 26557: (bug 23463 follow-up) Fix Batch import when incoming records contain itemnumber

Prior to ug 23463 AddItemFromMarc where calling AddItem, that did not
take into account the itemnumber field.
Now that we are using Koha::Item, we need to remove the items.itemnumber
field from the MARC record

Test plan:
1 - find an existing bib in your system with just one item
2 - export that bib with the item attached
3 - delete the barcode from your item in Koha
4 - stage your exported marc file for reimport, match on biblionumber, set it to Always Add Items
5 - confirm that the bib matches and the incoming 952 is parsed
6 - click "Import this batch into the catalog"

=> Without this patch you get (in the logs, or hidden)
manage-marc-import.pl: DBD::mysql::st execute failed: Duplicate entry '23' for key 'PRIMARY' [for Statement "INSERT INTO `items` ( `barcode`, `biblioitemnumber`, `biblionumber`, `ccode`, `cn_sort`, `cn_source`, `damaged_on`, `dateaccessioned`, `datelastborrowed`, `datelastseen`, `holdingbranch`, `homebranch`, `itemcallnumber`, `itemlost_on`, `itemnumber`, `itype`, `location`, `more_subfields_xml`, `onloan`, `permanent_location`, `replacementpricedate`, `timestamp`, `withdrawn_on`) VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, current_timestamp, ? )" with ParamValues: 0="BC_23", 1=8, 2=8, 3="REF", 4='CN__23', 5=undef, 6=undef, 7="2014-09-04", 8=undef, 9="2014-09-04", 10="FPL", 11="FPL", 12="CN_23", 13=undef, 14="23", 15="BK", 16="GEN", 17=undef, 18=undef, 19="GEN", 20="2014-09-04", 21=undef] at /usr/share/perl5/DBIx/Class/Storage/DBI.pm line 1836.
manage-marc-import.pl: DBIx::Class::Storage::DBI::_dbh_execute(): Duplicate entry '23' for key 'PRIMARY' at /kohadevbox/koha/Koha/Object.pm line 169
manage-marc-import.pl: {UNKNOWN}: Transaction aborted: Duplicate ID. Rollback failed: DBIx::Class::Storage::txn_rollback(): Refusing to roll back without a started transaction at /kohadevbox/koha/tools/manage-marc-import.pl line 253 at /kohadevbox/koha/tools/manage-marc-import.pl line 253

=> With this patch applied, the new item must be added to the existing bibliographic record

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 5 David Nind 2020-11-19 18:47:53 UTC
Created attachment 113845 [details] [review]
Bug 26557: (bug 23463 follow-up) Fix Batch import when incoming records contain itemnumber

Prior to ug 23463 AddItemFromMarc where calling AddItem, that did not
take into account the itemnumber field.
Now that we are using Koha::Item, we need to remove the items.itemnumber
field from the MARC record

Test plan:
1 - find an existing bib in your system with just one item
2 - export that bib with the item attached
3 - delete the barcode from your item in Koha
4 - stage your exported marc file for reimport, match on biblionumber, set it to Always Add Items
5 - confirm that the bib matches and the incoming 952 is parsed
6 - click "Import this batch into the catalog"

=> Without this patch you get (in the logs, or hidden)
manage-marc-import.pl: DBD::mysql::st execute failed: Duplicate entry '23' for key 'PRIMARY' [for Statement "INSERT INTO `items` ( `barcode`, `biblioitemnumber`, `biblionumber`, `ccode`, `cn_sort`, `cn_source`, `damaged_on`, `dateaccessioned`, `datelastborrowed`, `datelastseen`, `holdingbranch`, `homebranch`, `itemcallnumber`, `itemlost_on`, `itemnumber`, `itype`, `location`, `more_subfields_xml`, `onloan`, `permanent_location`, `replacementpricedate`, `timestamp`, `withdrawn_on`) VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, current_timestamp, ? )" with ParamValues: 0="BC_23", 1=8, 2=8, 3="REF", 4='CN__23', 5=undef, 6=undef, 7="2014-09-04", 8=undef, 9="2014-09-04", 10="FPL", 11="FPL", 12="CN_23", 13=undef, 14="23", 15="BK", 16="GEN", 17=undef, 18=undef, 19="GEN", 20="2014-09-04", 21=undef] at /usr/share/perl5/DBIx/Class/Storage/DBI.pm line 1836.
manage-marc-import.pl: DBIx::Class::Storage::DBI::_dbh_execute(): Duplicate entry '23' for key 'PRIMARY' at /kohadevbox/koha/Koha/Object.pm line 169
manage-marc-import.pl: {UNKNOWN}: Transaction aborted: Duplicate ID. Rollback failed: DBIx::Class::Storage::txn_rollback(): Refusing to roll back without a started transaction at /kohadevbox/koha/tools/manage-marc-import.pl line 253 at /kohadevbox/koha/tools/manage-marc-import.pl line 253

=> With this patch applied, the new item must be added to the existing bibliographic record

Signed-off-by: David Nind <david@davidnind.com>
Comment 6 Victor Grousset/tuxayo 2020-11-19 19:52:30 UTC
Sound like we both signed this off around the same time ^^

I'm resending the patch with both signoffs
Comment 7 Victor Grousset/tuxayo 2020-11-19 19:53:51 UTC
Created attachment 113848 [details] [review]
Bug 26557: (bug 23463 follow-up) Fix Batch import when incoming records contain itemnumber

Prior to ug 23463 AddItemFromMarc where calling AddItem, that did not
take into account the itemnumber field.
Now that we are using Koha::Item, we need to remove the items.itemnumber
field from the MARC record

Test plan:
1 - find an existing bib in your system with just one item
2 - export that bib with the item attached
3 - delete the barcode from your item in Koha
4 - stage your exported marc file for reimport, match on biblionumber, set it to Always Add Items
5 - confirm that the bib matches and the incoming 952 is parsed
6 - click "Import this batch into the catalog"

=> Without this patch you get (in the logs, or hidden)
manage-marc-import.pl: DBD::mysql::st execute failed: Duplicate entry '23' for key 'PRIMARY' [for Statement "INSERT INTO `items` ( `barcode`, `biblioitemnumber`, `biblionumber`, `ccode`, `cn_sort`, `cn_source`, `damaged_on`, `dateaccessioned`, `datelastborrowed`, `datelastseen`, `holdingbranch`, `homebranch`, `itemcallnumber`, `itemlost_on`, `itemnumber`, `itype`, `location`, `more_subfields_xml`, `onloan`, `permanent_location`, `replacementpricedate`, `timestamp`, `withdrawn_on`) VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, current_timestamp, ? )" with ParamValues: 0="BC_23", 1=8, 2=8, 3="REF", 4='CN__23', 5=undef, 6=undef, 7="2014-09-04", 8=undef, 9="2014-09-04", 10="FPL", 11="FPL", 12="CN_23", 13=undef, 14="23", 15="BK", 16="GEN", 17=undef, 18=undef, 19="GEN", 20="2014-09-04", 21=undef] at /usr/share/perl5/DBIx/Class/Storage/DBI.pm line 1836.
manage-marc-import.pl: DBIx::Class::Storage::DBI::_dbh_execute(): Duplicate entry '23' for key 'PRIMARY' at /kohadevbox/koha/Koha/Object.pm line 169
manage-marc-import.pl: {UNKNOWN}: Transaction aborted: Duplicate ID. Rollback failed: DBIx::Class::Storage::txn_rollback(): Refusing to roll back without a started transaction at /kohadevbox/koha/tools/manage-marc-import.pl line 253 at /kohadevbox/koha/tools/manage-marc-import.pl line 253

=> With this patch applied, the new item must be added to the existing bibliographic record

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 8 Martin Renvoize 2020-11-20 11:03:06 UTC
Created attachment 113871 [details] [review]
Bug 26557: (bug 23463 follow-up) Fix Batch import when incoming records contain itemnumber

Prior to ug 23463 AddItemFromMarc where calling AddItem, that did not
take into account the itemnumber field.
Now that we are using Koha::Item, we need to remove the items.itemnumber
field from the MARC record

Test plan:
1 - find an existing bib in your system with just one item
2 - export that bib with the item attached
3 - delete the barcode from your item in Koha
4 - stage your exported marc file for reimport, match on biblionumber, set it to Always Add Items
5 - confirm that the bib matches and the incoming 952 is parsed
6 - click "Import this batch into the catalog"

=> Without this patch you get (in the logs, or hidden)
manage-marc-import.pl: DBD::mysql::st execute failed: Duplicate entry '23' for key 'PRIMARY' [for Statement "INSERT INTO `items` ( `barcode`, `biblioitemnumber`, `biblionumber`, `ccode`, `cn_sort`, `cn_source`, `damaged_on`, `dateaccessioned`, `datelastborrowed`, `datelastseen`, `holdingbranch`, `homebranch`, `itemcallnumber`, `itemlost_on`, `itemnumber`, `itype`, `location`, `more_subfields_xml`, `onloan`, `permanent_location`, `replacementpricedate`, `timestamp`, `withdrawn_on`) VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, current_timestamp, ? )" with ParamValues: 0="BC_23", 1=8, 2=8, 3="REF", 4='CN__23', 5=undef, 6=undef, 7="2014-09-04", 8=undef, 9="2014-09-04", 10="FPL", 11="FPL", 12="CN_23", 13=undef, 14="23", 15="BK", 16="GEN", 17=undef, 18=undef, 19="GEN", 20="2014-09-04", 21=undef] at /usr/share/perl5/DBIx/Class/Storage/DBI.pm line 1836.
manage-marc-import.pl: DBIx::Class::Storage::DBI::_dbh_execute(): Duplicate entry '23' for key 'PRIMARY' at /kohadevbox/koha/Koha/Object.pm line 169
manage-marc-import.pl: {UNKNOWN}: Transaction aborted: Duplicate ID. Rollback failed: DBIx::Class::Storage::txn_rollback(): Refusing to roll back without a started transaction at /kohadevbox/koha/tools/manage-marc-import.pl line 253 at /kohadevbox/koha/tools/manage-marc-import.pl line 253

=> With this patch applied, the new item must be added to the existing bibliographic record

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 9 Martin Renvoize 2020-11-20 11:04:13 UTC
This works as expected, and the fix is clear.

I'm going to PQA, but would really like to see some Unit tests added in the future.. not blocking here in this case as tests need writing for nearly all the methods in this module so blocking this patch for lack of tests isn't pragmatic. :(
Comment 10 Jonathan Druart 2020-11-20 14:42:52 UTC
Pushed to master for 20.11, thanks to everybody involved!
Comment 11 Lucas Gass 2020-11-20 20:09:03 UTC
backported to 20.05.x for 20.05.06
Comment 12 Aleisha Amohia 2020-11-23 23:57:15 UTC
missing dependencies, not backported to 19.11.x