Description
Nick Clemens (kidclamp)
2023-02-02 18:57:04 UTC
Created attachment 146034 [details]
Sample file for importing with item from another record
Created attachment 146037 [details] [review] Bug 32804: Do not adjust biblionumber when replacing items during import This patch adjust the item matching at import to confirm that for a duplicate itemnumber or barcode matches an existing item in the DB and uses the original biblionumber when updating the item. When ordering in a consortium the items may be moved around, duplicate biblios added, and various matches found. We should not allow importing of items to move them from 1 biblio to another, but we should allow the imports to succeed and assume itemnumber or barcode matches are authoritative. The responsibility for correct matching of items to biblio should fall on the creator of the files To test: 1 - Be using the sample data in koha testing docker 2 - Stage the sample file on this report 3 - Match on KohaBiblio(999$c) / Item processing: replace 4 - Note the biblio match is "The complete novels / Austen, Jane" 5 - View the staged marc, note the barcode 39999000004090 in an item 6 - Search for that barcode 7 - You find biblio "Five novels : complete and unabridged / Gustave Flaubert" 8 - Import the file 9 - Check the db: SELECT * FROM items WHERE biblionumber != biblioitemnumber; 10 - Note the mismatch 11 - Fix the item and set it as 'Music' type UPDATE items SET biblionumber = biblioitemnumber, itype='MU' WHERE biblionumber != biblioitemnumber; 12 - Apply patch, restart all 13 - Stage and import the file with the same settings 14 - Confirm the item is modified on its original biblio (99) and that item type is Book again 15 - Change itemnumber to avoid itemnumber match and reset type UPDATE items SET itype='MU', itemnumber=999 WHERE itemnumber=212; 16 - Stage and import with the same setttings 17 - Confirm the marcode match worked and item is updated to book on original record 18 - Delete the original item 19 - Stage and import the file with the same settings 20 - The item is successfully created 21 - Stage and import, but item processing option is 'add' 22 - Confirm 1 item ignored 23 - Check the db SELECT * FROM import_items WHERE barcode=39999000004090 24 - Confirm there is a line with 'error' and duplicate_barcode At step 20 in the test plan I'm not getting an item created as described. I have deleted the item with barcode 39999000004090, but when I stage the batch set to match on biblionumber and replace items I don't get a new item created on import. The import_items table gives the error "duplicate item barcode" even though that barcode is in deleteditems and doesn't bring anything up in searches. Also, the behavior at steps 16/17 seems wrong to me. The incoming MARC has this item on one bib, Koha has it on another. To my mind, that suggests there's some fundamental confusion that should be sorted out manually. With this patch as written, the import and overlay edits an item on a bib that's not indicated in any way when looking at the batch, which seems like a good way miss errors in your import batch and to introduce unexpected item edits to your catalog. The item action choice is "Replace items when a matching bib is found." This is replacing an item on a non-matching bib. Created attachment 146292 [details] [review] Bug 32804: Add unit tests Created attachment 146293 [details] [review] Bug 32804: Do not adjust biblionumber when replacing items during import This patch adjust the item matching at import to confirm that for a duplicate itemnumber or barcode matches an existing item in the DB and uses the original biblionumber when updating the item. When ordering in a consortium the items may be moved around, duplicate biblios added, and various matches found. We should not allow importing of items to move them from 1 biblio to another, but we should allow the imports to succeed and assume itemnumber or barcode matches are authoritative. The responsibility for correct matching of items to biblio should fall on the creator of the files To test: 1 - Be using the sample data in koha testing docker 2 - Stage the sample file on this report 3 - Match on KohaBiblio(999$c) / Item processing: replace 4 - Note the biblio match is "The complete novels / Austen, Jane" 5 - View the staged marc, note the barcode 39999000004090 in an item 6 - Search for that barcode 7 - You find biblio "Five novels : complete and unabridged / Gustave Flaubert" 8 - Import the file 9 - Check the db: SELECT * FROM items WHERE biblionumber != biblioitemnumber; 10 - Note the mismatch 11 - Fix the item and set it as 'Music' type UPDATE items SET biblionumber = biblioitemnumber, itype='MU' WHERE biblionumber != biblioitemnumber; 12 - Apply patch, restart all 13 - Stage and import the file with the same settings 14 - Confirm the item is modified on its original biblio (99) and that item type is Book again 15 - Change itemnumber to avoid itemnumber match and reset type UPDATE items SET itype='MU', itemnumber=999 WHERE itemnumber=212; 16 - Stage and import with the same setttings 17 - Confirm the marcode match worked and item is updated to book on original record 18 - Delete the original item 19 - Stage and import the file with the same settings 20 - The item is successfully created 21 - Stage and import, but item processing option is 'add' 22 - Confirm 1 item ignored 23 - Check the db SELECT * FROM import_items WHERE barcode=39999000004090 24 - Confirm there is a line with 'error' and duplicate_barcode (In reply to Andrew Fuerste-Henry from comment #3) > At step 20 in the test plan I'm not getting an item created as described. I > have deleted the item with barcode 39999000004090, but when I stage the > batch set to match on biblionumber and replace items I don't get a new item > created on import. The import_items table gives the error "duplicate item > barcode" even though that barcode is in deleteditems and doesn't bring > anything up in searches. I did not have this experience when I attempted to reproduce the issue from your description. Everything worked as Nick described. > Also, the behavior at steps 16/17 seems wrong to me. The incoming MARC has > this item on one bib, Koha has it on another. To my mind, that suggests > there's some fundamental confusion that should be sorted out manually. With > this patch as written, the import and overlay edits an item on a bib that's > not indicated in any way when looking at the batch, which seems like a good > way miss errors in your import batch and to introduce unexpected item edits > to your catalog. The item action choice is "Replace items when a matching > bib is found." This is replacing an item on a non-matching bib. I can understand your pov, but I also think catalogers care less about which bib the item is attached to and much more about having the item overlay correctly. My experience is that they overlay the records and items, then merge all duplicate records. We may want to adjust the wording of that message to clarify that it overlays items if *any* record matches. Created attachment 146345 [details] [review] Bug 32804: Add unit tests Created attachment 146346 [details] [review] Bug 32804: Do not adjust biblionumber when replacing items during import This patch adjust the item matching at import to confirm that for a duplicate itemnumber or barcode matches an existing item in the DB and uses the original biblionumber when updating the item. When ordering in a consortium the items may be moved around, duplicate biblios added, and various matches found. We should not allow importing of items to move them from 1 biblio to another, but we should allow the imports to succeed and assume itemnumber or barcode matches are authoritative. The responsibility for correct matching of items to biblio should fall on the creator of the files To test: 1 - Be using the sample data in koha testing docker 2 - Stage the sample file on this report 3 - Match on KohaBiblio(999$c) / Item processing: replace 4 - Note the biblio match is "The complete novels / Austen, Jane" 5 - View the staged marc, note the barcode 39999000004090 in an item 6 - Search for that barcode 7 - You find biblio "Five novels : complete and unabridged / Gustave Flaubert" 8 - Import the file 9 - Check the db: SELECT * FROM items WHERE biblionumber != biblioitemnumber; 10 - Note the mismatch 11 - Fix the item and set it as 'Music' type UPDATE items SET biblionumber = biblioitemnumber, itype='MU' WHERE biblionumber != biblioitemnumber; 12 - Apply patch, restart all 13 - Stage and import the file with the same settings 14 - Confirm the item is modified on its original biblio (99) and that item type is Book again 15 - Change itemnumber to avoid itemnumber match and reset type UPDATE items SET itype='MU', itemnumber=999 WHERE itemnumber=212; 16 - Stage and import with the same setttings 17 - Confirm the marcode match worked and item is updated to book on original record 18 - Delete the original item 19 - Stage and import the file with the same settings 20 - The item is successfully created 21 - Stage and import, but item processing option is 'add' 22 - Confirm 1 item ignored 23 - Check the db SELECT * FROM import_items WHERE barcode=39999000004090 24 - Confirm there is a line with 'error' and duplicate_barcode (In reply to Kyle M Hall from comment #6) > (In reply to Andrew Fuerste-Henry from comment #3) > > At step 20 in the test plan I'm not getting an item created as described. I > > have deleted the item with barcode 39999000004090, but when I stage the > > batch set to match on biblionumber and replace items I don't get a new item > > created on import. The import_items table gives the error "duplicate item > > barcode" even though that barcode is in deleteditems and doesn't bring > > anything up in searches. > > I did not have this experience when I attempted to reproduce the issue from > your description. Everything worked as Nick described. I've just tested this again with the same result. The item with barcode 39999000004090 has been deleted but I get an import_error of "duplicate barcode' when importing the sample batch from this bug set to match on biblionumber and replace items. No new item is created. Having read through the test plan several times, I'm stumped. > I've just tested this again with the same result. The item with barcode
> 39999000004090 has been deleted but I get an import_error of "duplicate
> barcode' when importing the sample batch from this bug set to match on
> biblionumber and replace items. No new item is created. Having read through
> the test plan several times, I'm stumped.
I retested and I'm failing on step 20. Is that where you are having your issue as well?
(In reply to Kyle M Hall from comment #10) > > I've just tested this again with the same result. The item with barcode > > 39999000004090 has been deleted but I get an import_error of "duplicate > > barcode' when importing the sample batch from this bug set to match on > > biblionumber and replace items. No new item is created. Having read through > > the test plan several times, I'm stumped. > > I retested and I'm failing on step 20. Is that where you are having your > issue as well? Yes, at step 20 I don't get an item created. Created attachment 146579 [details] [review] Bug 32804: Do not adjust biblionumber when replacing items during import This patch adjust the item matching at import to confirm that for a duplicate itemnumber or barcode matches an existing item in the DB and uses the original biblionumber when updating the item. When ordering in a consortium the items may be moved around, duplicate biblios added, and various matches found. We should not allow importing of items to move them from 1 biblio to another, but we should allow the imports to succeed and assume itemnumber or barcode matches are authoritative. The responsibility for correct matching of items to biblio should fall on the creator of the files To test: 1 - Be using the sample data in koha testing docker 2 - Stage the sample file on this report 3 - Match on KohaBiblio(999$c) / Item processing: replace 4 - Note the biblio match is "The complete novels / Austen, Jane" 5 - View the staged marc, note the barcode 39999000004090 in an item 6 - Search for that barcode 7 - You find biblio "Five novels : complete and unabridged / Gustave Flaubert" 8 - Import the file 9 - Check the db: SELECT * FROM items WHERE biblionumber != biblioitemnumber; 10 - Note the mismatch 11 - Fix the item and set it as 'Music' type UPDATE items SET biblionumber = biblioitemnumber, itype='MU' WHERE biblionumber != biblioitemnumber; 12 - Apply patch, restart all 13 - Stage and import the file with the same settings 14 - Confirm the item is modified on its original biblio (99) and that item type is Book again 15 - Change itemnumber to avoid itemnumber match and reset type UPDATE items SET itype='MU', itemnumber=999 WHERE itemnumber=212; 16 - Stage and import with the same setttings 17 - Confirm the marcode match worked and item is updated to book on original record 18 - Delete the original item 19 - Stage and import the file with the same settings 20 - The item is successfully created 21 - Stage and import, but item processing option is 'add' 22 - Confirm 1 item ignored 23 - Check the db SELECT * FROM import_items WHERE barcode=39999000004090 24 - Confirm there is a line with 'error' and duplicate_barcode (In reply to Andrew Fuerste-Henry from comment #11) > (In reply to Kyle M Hall from comment #10) > > > I've just tested this again with the same result. The item with barcode > > > 39999000004090 has been deleted but I get an import_error of "duplicate > > > barcode' when importing the sample batch from this bug set to match on > > > biblionumber and replace items. No new item is created. Having read through > > > the test plan several times, I'm stumped. > > > > I retested and I'm failing on step 20. Is that where you are having your > > issue as well? > > Yes, at step 20 I don't get an item created. I had an uncommitted change on my local machine, patches updated, please test again Created attachment 147134 [details] [review] Bug 32804: Add unit tests JD amended patch - is( $item1->homebranch, $library->branchcode, "Item was overlayed succesfully" ); + is( $item1->homebranch, $library->branchcode, "Item was overlaid successfully" ); Created attachment 147135 [details] [review] Bug 32804: Do not adjust biblionumber when replacing items during import This patch adjust the item matching at import to confirm that for a duplicate itemnumber or barcode matches an existing item in the DB and uses the original biblionumber when updating the item. When ordering in a consortium the items may be moved around, duplicate biblios added, and various matches found. We should not allow importing of items to move them from 1 biblio to another, but we should allow the imports to succeed and assume itemnumber or barcode matches are authoritative. The responsibility for correct matching of items to biblio should fall on the creator of the files To test: 1 - Be using the sample data in koha testing docker 2 - Stage the sample file on this report 3 - Match on KohaBiblio(999$c) / Item processing: replace 4 - Note the biblio match is "The complete novels / Austen, Jane" 5 - View the staged marc, note the barcode 39999000004090 in an item 6 - Search for that barcode 7 - You find biblio "Five novels : complete and unabridged / Gustave Flaubert" 8 - Import the file 9 - Check the db: SELECT * FROM items WHERE biblionumber != biblioitemnumber; 10 - Note the mismatch 11 - Fix the item and set it as 'Music' type UPDATE items SET biblionumber = biblioitemnumber, itype='MU' WHERE biblionumber != biblioitemnumber; 12 - Apply patch, restart all 13 - Stage and import the file with the same settings 14 - Confirm the item is modified on its original biblio (99) and that item type is Book again 15 - Change itemnumber to avoid itemnumber match and reset type UPDATE items SET itype='MU', itemnumber=999 WHERE itemnumber=212; 16 - Stage and import with the same setttings 17 - Confirm the marcode match worked and item is updated to book on original record 18 - Delete the original item 19 - Stage and import the file with the same settings 20 - The item is successfully created 21 - Stage and import, but item processing option is 'add' 22 - Confirm 1 item ignored 23 - Check the db SELECT * FROM import_items WHERE barcode=39999000004090 24 - Confirm there is a line with 'error' and duplicate_barcode JD amended patch - # We assume that when replaicing tiems we do not want to move them - the onus is on the importer to + # We assume that when replacing items we do not want to move them - the onus is on the importer to Created attachment 147668 [details] [review] Bug 32804: Add unit tests JD amended patch - is( $item1->homebranch, $library->branchcode, "Item was overlayed succesfully" ); + is( $item1->homebranch, $library->branchcode, "Item was overlaid successfully" ); Signed-off-by: Andrew Fuerste-Henry <andrewfh@dubcolib.org> Created attachment 147669 [details] [review] Bug 32804: Do not adjust biblionumber when replacing items during import This patch adjust the item matching at import to confirm that for a duplicate itemnumber or barcode matches an existing item in the DB and uses the original biblionumber when updating the item. When ordering in a consortium the items may be moved around, duplicate biblios added, and various matches found. We should not allow importing of items to move them from 1 biblio to another, but we should allow the imports to succeed and assume itemnumber or barcode matches are authoritative. The responsibility for correct matching of items to biblio should fall on the creator of the files To test: 1 - Be using the sample data in koha testing docker 2 - Stage the sample file on this report 3 - Match on KohaBiblio(999$c) / Item processing: replace 4 - Note the biblio match is "The complete novels / Austen, Jane" 5 - View the staged marc, note the barcode 39999000004090 in an item 6 - Search for that barcode 7 - You find biblio "Five novels : complete and unabridged / Gustave Flaubert" 8 - Import the file 9 - Check the db: SELECT * FROM items WHERE biblionumber != biblioitemnumber; 10 - Note the mismatch 11 - Fix the item and set it as 'Music' type UPDATE items SET biblionumber = biblioitemnumber, itype='MU' WHERE biblionumber != biblioitemnumber; 12 - Apply patch, restart all 13 - Stage and import the file with the same settings 14 - Confirm the item is modified on its original biblio (99) and that item type is Book again 15 - Change itemnumber to avoid itemnumber match and reset type UPDATE items SET itype='MU', itemnumber=999 WHERE itemnumber=212; 16 - Stage and import with the same setttings 17 - Confirm the marcode match worked and item is updated to book on original record 18 - Delete the original item 19 - Stage and import the file with the same settings 20 - The item is successfully created 21 - Stage and import, but item processing option is 'add' 22 - Confirm 1 item ignored 23 - Check the db SELECT * FROM import_items WHERE barcode=39999000004090 24 - Confirm there is a line with 'error' and duplicate_barcode JD amended patch - # We assume that when replaicing tiems we do not want to move them - the onus is on the importer to + # We assume that when replacing items we do not want to move them - the onus is on the importer to Signed-off-by: Andrew Fuerste-Henry <andrewfh@dubcolib.org> # Subtest: Do not adjust biblionumber when replacing items during import 1..6 ok 1 - Item's homebranch is currently not the same as our created branch's branchcode ok 2 - Item was replaced ok 3 - Import was successful ok 4 - No error was reported not ok 5 - Item's biblionumber and biblioitemnumber match # Failed test 'Item's biblionumber and biblioitemnumber match' # at t/db_dependent/ImportBatch.t line 404. # # got: '3899' # expected: '2138' # Created attachment 147690 [details] [review] Bug 32804: (follow-up) Test that id's have not changed Rather than assuming biblionumber and biblioitemnumber are the same, we should test that they have not changed form their original values Looking here Created attachment 148314 [details] [review] Bug 32804: Add unit tests JD amended patch - is( $item1->homebranch, $library->branchcode, "Item was overlayed succesfully" ); + is( $item1->homebranch, $library->branchcode, "Item was overlaid successfully" ); Signed-off-by: Andrew Fuerste-Henry <andrewfh@dubcolib.org> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Created attachment 148315 [details] [review] Bug 32804: Do not adjust biblionumber when replacing items during import This patch adjust the item matching at import to confirm that for a duplicate itemnumber or barcode matches an existing item in the DB and uses the original biblionumber when updating the item. When ordering in a consortium the items may be moved around, duplicate biblios added, and various matches found. We should not allow importing of items to move them from 1 biblio to another, but we should allow the imports to succeed and assume itemnumber or barcode matches are authoritative. The responsibility for correct matching of items to biblio should fall on the creator of the files To test: 1 - Be using the sample data in koha testing docker 2 - Stage the sample file on this report 3 - Match on KohaBiblio(999$c) / Item processing: replace 4 - Note the biblio match is "The complete novels / Austen, Jane" 5 - View the staged marc, note the barcode 39999000004090 in an item 6 - Search for that barcode 7 - You find biblio "Five novels : complete and unabridged / Gustave Flaubert" 8 - Import the file 9 - Check the db: SELECT * FROM items WHERE biblionumber != biblioitemnumber; 10 - Note the mismatch 11 - Fix the item and set it as 'Music' type UPDATE items SET biblionumber = biblioitemnumber, itype='MU' WHERE biblionumber != biblioitemnumber; 12 - Apply patch, restart all 13 - Stage and import the file with the same settings 14 - Confirm the item is modified on its original biblio (99) and that item type is Book again 15 - Change itemnumber to avoid itemnumber match and reset type UPDATE items SET itype='MU', itemnumber=999 WHERE itemnumber=212; 16 - Stage and import with the same setttings 17 - Confirm the marcode match worked and item is updated to book on original record 18 - Delete the original item 19 - Stage and import the file with the same settings 20 - The item is successfully created 21 - Stage and import, but item processing option is 'add' 22 - Confirm 1 item ignored 23 - Check the db SELECT * FROM import_items WHERE barcode=39999000004090 24 - Confirm there is a line with 'error' and duplicate_barcode JD amended patch - # We assume that when replaicing tiems we do not want to move them - the onus is on the importer to + # We assume that when replacing items we do not want to move them - the onus is on the importer to Signed-off-by: Andrew Fuerste-Henry <andrewfh@dubcolib.org> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Created attachment 148316 [details] [review] Bug 32804: (follow-up) Test that id's have not changed Rather than assuming biblionumber and biblioitemnumber are the same, we should test that they have not changed form their original values Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Created attachment 148317 [details] [review] Bug 32804: (QA follow-up) Typo ahs and fix ImportBatch.t Resolve: # Failed test 'Item's biblioitemnumber has not changed' # at t/db_dependent/ImportBatch.t line 407. # # got: '4261' # expected: '2371' Do not compare $item1->biblionumber with $original_biblioitemnumber :) Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Pushed to master for 23.05. Nice work everyone, thanks! Nice work everyone! Pushed to stable for 22.11.x Merge conflicts with 22.05.x, no backport. If needed please rebase. Created attachment 150363 [details] [review] Bug 32804: [22.05.x] Do not adjust biblionumber when replacing items during import This patch adjust the item matching at import to confirm that for a duplicate itemnumber or barcode matches an existing item in the DB and uses the original biblionumber when updating the item. When ordering in a consortium the items may be moved around, duplicate biblios added, and various matches found. We should not allow importing of items to move them from 1 biblio to another, but we should allow the imports to succeed and assume itemnumber or barcode matches are authoritative. The responsibility for correct matching of items to biblio should fall on the creator of the files To test: 1 - Be using the sample data in koha testing docker 2 - Stage the sample file on this report 3 - Match on KohaBiblio(999$c) / Item processing: replace 4 - Note the biblio match is "The complete novels / Austen, Jane" 5 - View the staged marc, note the barcode 39999000004090 in an item 6 - Search for that barcode 7 - You find biblio "Five novels : complete and unabridged / Gustave Flaubert" 8 - Import the file 9 - Check the db: SELECT * FROM items WHERE biblionumber != biblioitemnumber; 10 - Note the mismatch 11 - Fix the item and set it as 'Music' type UPDATE items SET biblionumber = biblioitemnumber, itype='MU' WHERE biblionumber != biblioitemnumber; 12 - Apply patch, restart all 13 - Stage and import the file with the same settings 14 - Confirm the item is modified on its original biblio (99) and that item type is Book again 15 - Change itemnumber to avoid itemnumber match and reset type UPDATE items SET itype='MU', itemnumber=999 WHERE itemnumber=212; 16 - Stage and import with the same setttings 17 - Confirm the marcode match worked and item is updated to book on original record 18 - Delete the original item 19 - Stage and import the file with the same settings 20 - The item is successfully created 21 - Stage and import, but item processing option is 'add' 22 - Confirm 1 item ignored 23 - Check the db SELECT * FROM import_items WHERE barcode=39999000004090 24 - Confirm there is a line with 'error' and duplicate_barcode JD amended patch - # We assume that when replaicing tiems we do not want to move them - the onus is on the importer to + # We assume that when replacing items we do not want to move them - the onus is on the importer to Signed-off-by: Andrew Fuerste-Henry <andrewfh@dubcolib.org> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Bug 32804: Add unit tests JD amended patch - is( $item1->homebranch, $library->branchcode, "Item was overlayed succesfully" ); + is( $item1->homebranch, $library->branchcode, "Item was overlaid successfully" ); Signed-off-by: Andrew Fuerste-Henry <andrewfh@dubcolib.org> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Bug 32804: (follow-up) Test that id's have not changed Rather than assuming biblionumber and biblioitemnumber are the same, we should test that they have not changed form their original values Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Bug 32804: (QA follow-up) Typo ahs and fix ImportBatch.t Resolve: # Failed test 'Item's biblioitemnumber has not changed' # at t/db_dependent/ImportBatch.t line 407. # # got: '4261' # expected: '2371' Do not compare $item1->biblionumber with $original_biblioitemnumber :) Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> (In reply to Lucas Gass from comment #27) > Merge conflicts with 22.05.x, no backport. If needed please rebase. Patch rebased, please test before pushing |