Bug 32804 - Importing and replacing items causes inconsistency when itemnumber match and biblio match disagree
Summary: Importing and replacing items causes inconsistency when itemnumber match and ...
Status: RESOLVED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Tools (show other bugs)
Version: Main
Hardware: All All
: P5 - low critical (vote)
Assignee: Nick Clemens
QA Contact: Marcel de Rooy
URL:
Keywords:
Depends on:
Blocks: 32901
  Show dependency treegraph
 
Reported: 2023-02-02 18:57 UTC by Nick Clemens
Modified: 2023-12-28 20:44 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
23.05.00,22.11.04


Attachments
Sample file for importing with item from another record (5.70 KB, application/xml)
2023-02-02 20:16 UTC, Nick Clemens
Details
Bug 32804: Do not adjust biblionumber when replacing items during import (4.77 KB, patch)
2023-02-02 20:34 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 32804: Add unit tests (2.74 KB, patch)
2023-02-06 17:10 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 32804: Do not adjust biblionumber when replacing items during import (4.77 KB, patch)
2023-02-06 17:10 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 32804: Add unit tests (2.72 KB, patch)
2023-02-07 17:11 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 32804: Do not adjust biblionumber when replacing items during import (4.77 KB, patch)
2023-02-07 17:11 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 32804: Do not adjust biblionumber when replacing items during import (5.36 KB, patch)
2023-02-13 13:25 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 32804: Add unit tests (2.91 KB, patch)
2023-02-22 13:00 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 32804: Do not adjust biblionumber when replacing items during import (5.59 KB, patch)
2023-02-22 13:00 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 32804: Add unit tests (2.97 KB, patch)
2023-03-02 18:44 UTC, Andrew Fuerste-Henry
Details | Diff | Splinter Review
Bug 32804: Do not adjust biblionumber when replacing items during import (5.65 KB, patch)
2023-03-02 18:44 UTC, Andrew Fuerste-Henry
Details | Diff | Splinter Review
Bug 32804: (follow-up) Test that id's have not changed (1.76 KB, patch)
2023-03-03 12:20 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 32804: Add unit tests (3.06 KB, patch)
2023-03-17 09:14 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 32804: Do not adjust biblionumber when replacing items during import (5.75 KB, patch)
2023-03-17 09:14 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 32804: (follow-up) Test that id's have not changed (1.85 KB, patch)
2023-03-17 09:14 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 32804: (QA follow-up) Typo ahs and fix ImportBatch.t (2.03 KB, patch)
2023-03-17 09:14 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 32804: [22.05.x] Do not adjust biblionumber when replacing items during import (9.17 KB, patch)
2023-04-28 11:02 UTC, Nick Clemens
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Nick Clemens 2023-02-02 18:57:04 UTC
When 'Item processing' option during import is set to 'replace' itemnumber matches do not check any other data - so the item's biblionumber is set to the matched biblionumber, but the biblioitemnumber is not updated or checked

This means that if you import a record containing an item with an itemnumber from another record you will end up with an item with mismatched biblio and biblioitem numbers

This happens in consortia when multiple libraries have ordered the same biblio. Two biblios are created, later the final import with barcodes etc. is imported to replace items, and the more full bib is chosen for the biblio match, so items are half moved.

To recreate:
1 - Be using the sample data in koha testing docker
2 - Stage the sample file on this report
3 - Match on KohaBiblio(999$c) / Item processing: replace
4 - Note the biblio match is "The complete novels / Austen, Jane"
5 - View the staged marc, note the barcode 39999000004090 in an item
6 - Search for that barcode
7 - You find biblio "Five novels : complete and unabridged / Gustave Flaubert"
8 - Import the file
9 - Check the db:
    SELECT * FROM items WHERE biblionumber != biblioitemnumber;
10 - Note the mismatch
Comment 1 Nick Clemens 2023-02-02 20:16:31 UTC
Created attachment 146034 [details]
Sample file for importing with item from another record
Comment 2 Nick Clemens 2023-02-02 20:34:07 UTC
Created attachment 146037 [details] [review]
Bug 32804: Do not adjust biblionumber when replacing items during import

This patch adjust the item matching at import to confirm that for a duplicate itemnumber or barcode
matches an existing item in the DB and uses the original biblionumber when updating the item.

When ordering in a consortium the items may be moved around, duplicate biblios added, and various matches found.
We should not allow importing of items to move them from 1 biblio to another, but  we should allow the imports
to succeed and assume itemnumber or barcode matches are authoritative. The responsibility for correct matching of items to biblio should fall on the creator of the files

To test:
 1 - Be using the sample data in koha testing docker
 2 - Stage the sample file on this report
 3 - Match on KohaBiblio(999$c) / Item processing: replace
 4 - Note the biblio match is "The complete novels / Austen, Jane"
 5 - View the staged marc, note the barcode 39999000004090 in an item
 6 - Search for that barcode
 7 - You find biblio "Five novels : complete and unabridged / Gustave Flaubert"
 8 - Import the file
 9 - Check the db:
     SELECT * FROM items WHERE biblionumber != biblioitemnumber;
10 - Note the mismatch
11 - Fix the item and set it as 'Music' type
     UPDATE items SET biblionumber = biblioitemnumber, itype='MU' WHERE biblionumber != biblioitemnumber;
12 - Apply patch, restart all
13 - Stage and import the file with the same settings
14 - Confirm the item is modified on its original biblio (99) and that item type is Book again
15 - Change itemnumber to avoid itemnumber match and reset type
     UPDATE items SET itype='MU', itemnumber=999 WHERE itemnumber=212;
16 - Stage and import with the same setttings
17 - Confirm the marcode match worked and item is updated to book on original record
18 - Delete the original item
19 - Stage and import the file with the same settings
20 - The item is successfully created
21 - Stage and import, but item processing option is 'add'
22 - Confirm 1 item ignored
23 - Check the db
     SELECT * FROM import_items WHERE barcode=39999000004090
24 - Confirm there is a line with 'error' and duplicate_barcode
Comment 3 Andrew Fuerste-Henry 2023-02-03 15:46:09 UTC
At step 20 in the test plan I'm not getting an item created as described. I have deleted the item with barcode 39999000004090, but when I stage the batch set to match on biblionumber and replace items I don't get a new item created on import. The import_items table gives the error "duplicate item barcode" even though that barcode is in deleteditems and doesn't bring anything up in searches.

Also, the behavior at steps 16/17 seems wrong to me. The incoming MARC has this item on one bib, Koha has it on another. To my mind, that suggests there's some fundamental confusion that should be sorted out manually. With this patch as written, the import and overlay edits an item on a bib that's not indicated in any way when looking at the batch, which seems like a good way miss errors in your import batch and to introduce unexpected item edits to your catalog. The item action choice is "Replace items when a matching bib is found." This is replacing an item on a non-matching bib.
Comment 4 Kyle M Hall 2023-02-06 17:10:33 UTC
Created attachment 146292 [details] [review]
Bug 32804: Add unit tests
Comment 5 Kyle M Hall 2023-02-06 17:10:45 UTC
Created attachment 146293 [details] [review]
Bug 32804: Do not adjust biblionumber when replacing items during import

This patch adjust the item matching at import to confirm that for a duplicate itemnumber or barcode
matches an existing item in the DB and uses the original biblionumber when updating the item.

When ordering in a consortium the items may be moved around, duplicate biblios added, and various matches found.
We should not allow importing of items to move them from 1 biblio to another, but  we should allow the imports
to succeed and assume itemnumber or barcode matches are authoritative. The responsibility for correct matching of items to biblio should fall on the creator of the files

To test:
 1 - Be using the sample data in koha testing docker
 2 - Stage the sample file on this report
 3 - Match on KohaBiblio(999$c) / Item processing: replace
 4 - Note the biblio match is "The complete novels / Austen, Jane"
 5 - View the staged marc, note the barcode 39999000004090 in an item
 6 - Search for that barcode
 7 - You find biblio "Five novels : complete and unabridged / Gustave Flaubert"
 8 - Import the file
 9 - Check the db:
     SELECT * FROM items WHERE biblionumber != biblioitemnumber;
10 - Note the mismatch
11 - Fix the item and set it as 'Music' type
     UPDATE items SET biblionumber = biblioitemnumber, itype='MU' WHERE biblionumber != biblioitemnumber;
12 - Apply patch, restart all
13 - Stage and import the file with the same settings
14 - Confirm the item is modified on its original biblio (99) and that item type is Book again
15 - Change itemnumber to avoid itemnumber match and reset type
     UPDATE items SET itype='MU', itemnumber=999 WHERE itemnumber=212;
16 - Stage and import with the same setttings
17 - Confirm the marcode match worked and item is updated to book on original record
18 - Delete the original item
19 - Stage and import the file with the same settings
20 - The item is successfully created
21 - Stage and import, but item processing option is 'add'
22 - Confirm 1 item ignored
23 - Check the db
     SELECT * FROM import_items WHERE barcode=39999000004090
24 - Confirm there is a line with 'error' and duplicate_barcode
Comment 6 Kyle M Hall 2023-02-07 17:07:24 UTC
(In reply to Andrew Fuerste-Henry from comment #3)
> At step 20 in the test plan I'm not getting an item created as described. I
> have deleted the item with barcode 39999000004090, but when I stage the
> batch set to match on biblionumber and replace items I don't get a new item
> created on import. The import_items table gives the error "duplicate item
> barcode" even though that barcode is in deleteditems and doesn't bring
> anything up in searches.

I did not have this experience when I attempted to reproduce the issue from your description. Everything worked as Nick described.

> Also, the behavior at steps 16/17 seems wrong to me. The incoming MARC has
> this item on one bib, Koha has it on another. To my mind, that suggests
> there's some fundamental confusion that should be sorted out manually. With
> this patch as written, the import and overlay edits an item on a bib that's
> not indicated in any way when looking at the batch, which seems like a good
> way miss errors in your import batch and to introduce unexpected item edits
> to your catalog. The item action choice is "Replace items when a matching
> bib is found." This is replacing an item on a non-matching bib.

I can understand your pov, but I also think catalogers care less about which bib the item is attached to and much more about having the item overlay correctly. My experience is that they overlay the records and items, then merge all duplicate records. We may want to adjust the wording of that message to clarify that it overlays items if *any* record matches.
Comment 7 Kyle M Hall 2023-02-07 17:11:11 UTC
Created attachment 146345 [details] [review]
Bug 32804: Add unit tests
Comment 8 Kyle M Hall 2023-02-07 17:11:19 UTC
Created attachment 146346 [details] [review]
Bug 32804: Do not adjust biblionumber when replacing items during import

This patch adjust the item matching at import to confirm that for a duplicate itemnumber or barcode
matches an existing item in the DB and uses the original biblionumber when updating the item.

When ordering in a consortium the items may be moved around, duplicate biblios added, and various matches found.
We should not allow importing of items to move them from 1 biblio to another, but  we should allow the imports
to succeed and assume itemnumber or barcode matches are authoritative. The responsibility for correct matching of items to biblio should fall on the creator of the files

To test:
 1 - Be using the sample data in koha testing docker
 2 - Stage the sample file on this report
 3 - Match on KohaBiblio(999$c) / Item processing: replace
 4 - Note the biblio match is "The complete novels / Austen, Jane"
 5 - View the staged marc, note the barcode 39999000004090 in an item
 6 - Search for that barcode
 7 - You find biblio "Five novels : complete and unabridged / Gustave Flaubert"
 8 - Import the file
 9 - Check the db:
     SELECT * FROM items WHERE biblionumber != biblioitemnumber;
10 - Note the mismatch
11 - Fix the item and set it as 'Music' type
     UPDATE items SET biblionumber = biblioitemnumber, itype='MU' WHERE biblionumber != biblioitemnumber;
12 - Apply patch, restart all
13 - Stage and import the file with the same settings
14 - Confirm the item is modified on its original biblio (99) and that item type is Book again
15 - Change itemnumber to avoid itemnumber match and reset type
     UPDATE items SET itype='MU', itemnumber=999 WHERE itemnumber=212;
16 - Stage and import with the same setttings
17 - Confirm the marcode match worked and item is updated to book on original record
18 - Delete the original item
19 - Stage and import the file with the same settings
20 - The item is successfully created
21 - Stage and import, but item processing option is 'add'
22 - Confirm 1 item ignored
23 - Check the db
     SELECT * FROM import_items WHERE barcode=39999000004090
24 - Confirm there is a line with 'error' and duplicate_barcode
Comment 9 Andrew Fuerste-Henry 2023-02-08 20:15:50 UTC
(In reply to Kyle M Hall from comment #6)
> (In reply to Andrew Fuerste-Henry from comment #3)
> > At step 20 in the test plan I'm not getting an item created as described. I
> > have deleted the item with barcode 39999000004090, but when I stage the
> > batch set to match on biblionumber and replace items I don't get a new item
> > created on import. The import_items table gives the error "duplicate item
> > barcode" even though that barcode is in deleteditems and doesn't bring
> > anything up in searches.
> 
> I did not have this experience when I attempted to reproduce the issue from
> your description. Everything worked as Nick described.

I've just tested this again with the same result. The item with barcode 39999000004090 has been deleted but I get an import_error of "duplicate barcode' when importing the sample batch from this bug set to match on biblionumber and replace items. No new item is created. Having read through the test plan several times, I'm stumped.
Comment 10 Kyle M Hall 2023-02-09 18:24:21 UTC
> I've just tested this again with the same result. The item with barcode
> 39999000004090 has been deleted but I get an import_error of "duplicate
> barcode' when importing the sample batch from this bug set to match on
> biblionumber and replace items. No new item is created. Having read through
> the test plan several times, I'm stumped.

I retested and I'm failing on step 20. Is that where you are having your issue as well?
Comment 11 Andrew Fuerste-Henry 2023-02-09 18:30:57 UTC
(In reply to Kyle M Hall from comment #10)
> > I've just tested this again with the same result. The item with barcode
> > 39999000004090 has been deleted but I get an import_error of "duplicate
> > barcode' when importing the sample batch from this bug set to match on
> > biblionumber and replace items. No new item is created. Having read through
> > the test plan several times, I'm stumped.
> 
> I retested and I'm failing on step 20. Is that where you are having your
> issue as well?

Yes, at step 20 I don't get an item created.
Comment 12 Nick Clemens 2023-02-13 13:25:51 UTC
Created attachment 146579 [details] [review]
Bug 32804: Do not adjust biblionumber when replacing items during import

This patch adjust the item matching at import to confirm that for a duplicate itemnumber or barcode
matches an existing item in the DB and uses the original biblionumber when updating the item.

When ordering in a consortium the items may be moved around, duplicate biblios added, and various matches found.
We should not allow importing of items to move them from 1 biblio to another, but  we should allow the imports
to succeed and assume itemnumber or barcode matches are authoritative. The responsibility for correct matching of items to biblio should fall on the creator of the files

To test:
 1 - Be using the sample data in koha testing docker
 2 - Stage the sample file on this report
 3 - Match on KohaBiblio(999$c) / Item processing: replace
 4 - Note the biblio match is "The complete novels / Austen, Jane"
 5 - View the staged marc, note the barcode 39999000004090 in an item
 6 - Search for that barcode
 7 - You find biblio "Five novels : complete and unabridged / Gustave Flaubert"
 8 - Import the file
 9 - Check the db:
     SELECT * FROM items WHERE biblionumber != biblioitemnumber;
10 - Note the mismatch
11 - Fix the item and set it as 'Music' type
     UPDATE items SET biblionumber = biblioitemnumber, itype='MU' WHERE biblionumber != biblioitemnumber;
12 - Apply patch, restart all
13 - Stage and import the file with the same settings
14 - Confirm the item is modified on its original biblio (99) and that item type is Book again
15 - Change itemnumber to avoid itemnumber match and reset type
     UPDATE items SET itype='MU', itemnumber=999 WHERE itemnumber=212;
16 - Stage and import with the same setttings
17 - Confirm the marcode match worked and item is updated to book on original record
18 - Delete the original item
19 - Stage and import the file with the same settings
20 - The item is successfully created
21 - Stage and import, but item processing option is 'add'
22 - Confirm 1 item ignored
23 - Check the db
     SELECT * FROM import_items WHERE barcode=39999000004090
24 - Confirm there is a line with 'error' and duplicate_barcode
Comment 13 Nick Clemens 2023-02-13 13:26:22 UTC
(In reply to Andrew Fuerste-Henry from comment #11)
> (In reply to Kyle M Hall from comment #10)
> > > I've just tested this again with the same result. The item with barcode
> > > 39999000004090 has been deleted but I get an import_error of "duplicate
> > > barcode' when importing the sample batch from this bug set to match on
> > > biblionumber and replace items. No new item is created. Having read through
> > > the test plan several times, I'm stumped.
> > 
> > I retested and I'm failing on step 20. Is that where you are having your
> > issue as well?
> 
> Yes, at step 20 I don't get an item created.

I had an uncommitted change on my local machine, patches updated, please test again
Comment 14 Jonathan Druart 2023-02-22 13:00:08 UTC
Created attachment 147134 [details] [review]
Bug 32804: Add unit tests

JD amended patch
-    is( $item1->homebranch, $library->branchcode, "Item was overlayed succesfully" );
+    is( $item1->homebranch, $library->branchcode, "Item was overlaid successfully" );
Comment 15 Jonathan Druart 2023-02-22 13:00:11 UTC
Created attachment 147135 [details] [review]
Bug 32804: Do not adjust biblionumber when replacing items during import

This patch adjust the item matching at import to confirm that for a duplicate itemnumber or barcode
matches an existing item in the DB and uses the original biblionumber when updating the item.

When ordering in a consortium the items may be moved around, duplicate biblios added, and various matches found.
We should not allow importing of items to move them from 1 biblio to another, but  we should allow the imports
to succeed and assume itemnumber or barcode matches are authoritative. The responsibility for correct matching of items to biblio should fall on the creator of the files

To test:
 1 - Be using the sample data in koha testing docker
 2 - Stage the sample file on this report
 3 - Match on KohaBiblio(999$c) / Item processing: replace
 4 - Note the biblio match is "The complete novels / Austen, Jane"
 5 - View the staged marc, note the barcode 39999000004090 in an item
 6 - Search for that barcode
 7 - You find biblio "Five novels : complete and unabridged / Gustave Flaubert"
 8 - Import the file
 9 - Check the db:
     SELECT * FROM items WHERE biblionumber != biblioitemnumber;
10 - Note the mismatch
11 - Fix the item and set it as 'Music' type
     UPDATE items SET biblionumber = biblioitemnumber, itype='MU' WHERE biblionumber != biblioitemnumber;
12 - Apply patch, restart all
13 - Stage and import the file with the same settings
14 - Confirm the item is modified on its original biblio (99) and that item type is Book again
15 - Change itemnumber to avoid itemnumber match and reset type
     UPDATE items SET itype='MU', itemnumber=999 WHERE itemnumber=212;
16 - Stage and import with the same setttings
17 - Confirm the marcode match worked and item is updated to book on original record
18 - Delete the original item
19 - Stage and import the file with the same settings
20 - The item is successfully created
21 - Stage and import, but item processing option is 'add'
22 - Confirm 1 item ignored
23 - Check the db
     SELECT * FROM import_items WHERE barcode=39999000004090
24 - Confirm there is a line with 'error' and duplicate_barcode

JD amended patch
-        # We assume that when replaicing tiems we do not want to move them - the onus is on the importer to
+        # We assume that when replacing items we do not want to move them - the onus is on the importer to
Comment 16 Andrew Fuerste-Henry 2023-03-02 18:44:27 UTC
Created attachment 147668 [details] [review]
Bug 32804: Add unit tests

JD amended patch
-    is( $item1->homebranch, $library->branchcode, "Item was overlayed succesfully" );
+    is( $item1->homebranch, $library->branchcode, "Item was overlaid successfully" );

Signed-off-by: Andrew Fuerste-Henry <andrewfh@dubcolib.org>
Comment 17 Andrew Fuerste-Henry 2023-03-02 18:44:29 UTC
Created attachment 147669 [details] [review]
Bug 32804: Do not adjust biblionumber when replacing items during import

This patch adjust the item matching at import to confirm that for a duplicate itemnumber or barcode
matches an existing item in the DB and uses the original biblionumber when updating the item.

When ordering in a consortium the items may be moved around, duplicate biblios added, and various matches found.
We should not allow importing of items to move them from 1 biblio to another, but  we should allow the imports
to succeed and assume itemnumber or barcode matches are authoritative. The responsibility for correct matching of items to biblio should fall on the creator of the files

To test:
 1 - Be using the sample data in koha testing docker
 2 - Stage the sample file on this report
 3 - Match on KohaBiblio(999$c) / Item processing: replace
 4 - Note the biblio match is "The complete novels / Austen, Jane"
 5 - View the staged marc, note the barcode 39999000004090 in an item
 6 - Search for that barcode
 7 - You find biblio "Five novels : complete and unabridged / Gustave Flaubert"
 8 - Import the file
 9 - Check the db:
     SELECT * FROM items WHERE biblionumber != biblioitemnumber;
10 - Note the mismatch
11 - Fix the item and set it as 'Music' type
     UPDATE items SET biblionumber = biblioitemnumber, itype='MU' WHERE biblionumber != biblioitemnumber;
12 - Apply patch, restart all
13 - Stage and import the file with the same settings
14 - Confirm the item is modified on its original biblio (99) and that item type is Book again
15 - Change itemnumber to avoid itemnumber match and reset type
     UPDATE items SET itype='MU', itemnumber=999 WHERE itemnumber=212;
16 - Stage and import with the same setttings
17 - Confirm the marcode match worked and item is updated to book on original record
18 - Delete the original item
19 - Stage and import the file with the same settings
20 - The item is successfully created
21 - Stage and import, but item processing option is 'add'
22 - Confirm 1 item ignored
23 - Check the db
     SELECT * FROM import_items WHERE barcode=39999000004090
24 - Confirm there is a line with 'error' and duplicate_barcode

JD amended patch
-        # We assume that when replaicing tiems we do not want to move them - the onus is on the importer to
+        # We assume that when replacing items we do not want to move them - the onus is on the importer to

Signed-off-by: Andrew Fuerste-Henry <andrewfh@dubcolib.org>
Comment 18 Marcel de Rooy 2023-03-03 10:22:59 UTC
# Subtest: Do not adjust biblionumber when replacing items during import
    1..6
    ok 1 - Item's homebranch is currently not the same as our created branch's branchcode
    ok 2 - Item was replaced
    ok 3 - Import was successful
    ok 4 - No error was reported
    not ok 5 - Item's biblionumber and biblioitemnumber match
    #   Failed test 'Item's biblionumber and biblioitemnumber match'
    #   at t/db_dependent/ImportBatch.t line 404.
    #
    #          got: '3899'
    #     expected: '2138'
    #
Comment 19 Nick Clemens 2023-03-03 12:20:08 UTC
Created attachment 147690 [details] [review]
Bug 32804: (follow-up) Test that id's have not changed

Rather than assuming biblionumber and biblioitemnumber are the same, we
should test that they have not changed form their original values
Comment 20 Marcel de Rooy 2023-03-17 08:31:03 UTC
Looking here
Comment 21 Marcel de Rooy 2023-03-17 09:14:14 UTC
Created attachment 148314 [details] [review]
Bug 32804: Add unit tests

JD amended patch
-    is( $item1->homebranch, $library->branchcode, "Item was overlayed succesfully" );
+    is( $item1->homebranch, $library->branchcode, "Item was overlaid successfully" );

Signed-off-by: Andrew Fuerste-Henry <andrewfh@dubcolib.org>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 22 Marcel de Rooy 2023-03-17 09:14:17 UTC
Created attachment 148315 [details] [review]
Bug 32804: Do not adjust biblionumber when replacing items during import

This patch adjust the item matching at import to confirm that for a duplicate itemnumber or barcode
matches an existing item in the DB and uses the original biblionumber when updating the item.

When ordering in a consortium the items may be moved around, duplicate biblios added, and various matches found.
We should not allow importing of items to move them from 1 biblio to another, but  we should allow the imports
to succeed and assume itemnumber or barcode matches are authoritative. The responsibility for correct matching of items to biblio should fall on the creator of the files

To test:
 1 - Be using the sample data in koha testing docker
 2 - Stage the sample file on this report
 3 - Match on KohaBiblio(999$c) / Item processing: replace
 4 - Note the biblio match is "The complete novels / Austen, Jane"
 5 - View the staged marc, note the barcode 39999000004090 in an item
 6 - Search for that barcode
 7 - You find biblio "Five novels : complete and unabridged / Gustave Flaubert"
 8 - Import the file
 9 - Check the db:
     SELECT * FROM items WHERE biblionumber != biblioitemnumber;
10 - Note the mismatch
11 - Fix the item and set it as 'Music' type
     UPDATE items SET biblionumber = biblioitemnumber, itype='MU' WHERE biblionumber != biblioitemnumber;
12 - Apply patch, restart all
13 - Stage and import the file with the same settings
14 - Confirm the item is modified on its original biblio (99) and that item type is Book again
15 - Change itemnumber to avoid itemnumber match and reset type
     UPDATE items SET itype='MU', itemnumber=999 WHERE itemnumber=212;
16 - Stage and import with the same setttings
17 - Confirm the marcode match worked and item is updated to book on original record
18 - Delete the original item
19 - Stage and import the file with the same settings
20 - The item is successfully created
21 - Stage and import, but item processing option is 'add'
22 - Confirm 1 item ignored
23 - Check the db
     SELECT * FROM import_items WHERE barcode=39999000004090
24 - Confirm there is a line with 'error' and duplicate_barcode

JD amended patch
-        # We assume that when replaicing tiems we do not want to move them - the onus is on the importer to
+        # We assume that when replacing items we do not want to move them - the onus is on the importer to

Signed-off-by: Andrew Fuerste-Henry <andrewfh@dubcolib.org>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 23 Marcel de Rooy 2023-03-17 09:14:19 UTC
Created attachment 148316 [details] [review]
Bug 32804: (follow-up) Test that id's have not changed

Rather than assuming biblionumber and biblioitemnumber are the same, we
should test that they have not changed form their original values

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 24 Marcel de Rooy 2023-03-17 09:14:22 UTC
Created attachment 148317 [details] [review]
Bug 32804: (QA follow-up) Typo ahs and fix ImportBatch.t

Resolve:
    #   Failed test 'Item's biblioitemnumber has not changed'
    #   at t/db_dependent/ImportBatch.t line 407.
    #
    #          got: '4261'
    #     expected: '2371'

Do not compare $item1->biblionumber with $original_biblioitemnumber :)

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 25 Tomás Cohen Arazi 2023-03-17 12:59:51 UTC
Pushed to master for 23.05.

Nice work everyone, thanks!
Comment 26 Matt Blenkinsop 2023-03-17 17:20:54 UTC
Nice work everyone!

Pushed to stable for 22.11.x
Comment 27 Lucas Gass 2023-04-24 16:31:39 UTC
Merge conflicts with 22.05.x, no backport. If needed please rebase.
Comment 28 Nick Clemens 2023-04-28 11:02:12 UTC
Created attachment 150363 [details] [review]
Bug 32804: [22.05.x] Do not adjust biblionumber when replacing items during import

This patch adjust the item matching at import to confirm that for a duplicate itemnumber or barcode
matches an existing item in the DB and uses the original biblionumber when updating the item.

When ordering in a consortium the items may be moved around, duplicate biblios added, and various matches found.
We should not allow importing of items to move them from 1 biblio to another, but  we should allow the imports
to succeed and assume itemnumber or barcode matches are authoritative. The responsibility for correct matching of items to biblio should fall on the creator of the files

To test:
 1 - Be using the sample data in koha testing docker
 2 - Stage the sample file on this report
 3 - Match on KohaBiblio(999$c) / Item processing: replace
 4 - Note the biblio match is "The complete novels / Austen, Jane"
 5 - View the staged marc, note the barcode 39999000004090 in an item
 6 - Search for that barcode
 7 - You find biblio "Five novels : complete and unabridged / Gustave Flaubert"
 8 - Import the file
 9 - Check the db:
     SELECT * FROM items WHERE biblionumber != biblioitemnumber;
10 - Note the mismatch
11 - Fix the item and set it as 'Music' type
     UPDATE items SET biblionumber = biblioitemnumber, itype='MU' WHERE biblionumber != biblioitemnumber;
12 - Apply patch, restart all
13 - Stage and import the file with the same settings
14 - Confirm the item is modified on its original biblio (99) and that item type is Book again
15 - Change itemnumber to avoid itemnumber match and reset type
     UPDATE items SET itype='MU', itemnumber=999 WHERE itemnumber=212;
16 - Stage and import with the same setttings
17 - Confirm the marcode match worked and item is updated to book on original record
18 - Delete the original item
19 - Stage and import the file with the same settings
20 - The item is successfully created
21 - Stage and import, but item processing option is 'add'
22 - Confirm 1 item ignored
23 - Check the db
     SELECT * FROM import_items WHERE barcode=39999000004090
24 - Confirm there is a line with 'error' and duplicate_barcode

JD amended patch
-        # We assume that when replaicing tiems we do not want to move them - the onus is on the importer to
+        # We assume that when replacing items we do not want to move them - the onus is on the importer to

Signed-off-by: Andrew Fuerste-Henry <andrewfh@dubcolib.org>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Bug 32804: Add unit tests

JD amended patch
-    is( $item1->homebranch, $library->branchcode, "Item was overlayed succesfully" );
+    is( $item1->homebranch, $library->branchcode, "Item was overlaid successfully" );

Signed-off-by: Andrew Fuerste-Henry <andrewfh@dubcolib.org>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Bug 32804: (follow-up) Test that id's have not changed

Rather than assuming biblionumber and biblioitemnumber are the same, we
should test that they have not changed form their original values

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Bug 32804: (QA follow-up) Typo ahs and fix ImportBatch.t

Resolve:
    #   Failed test 'Item's biblioitemnumber has not changed'
    #   at t/db_dependent/ImportBatch.t line 407.
    #
    #          got: '4261'
    #     expected: '2371'

Do not compare $item1->biblionumber with $original_biblioitemnumber :)

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 29 Nick Clemens 2023-04-28 11:02:45 UTC
(In reply to Lucas Gass from comment #27)
> Merge conflicts with 22.05.x, no backport. If needed please rebase.

Patch rebased, please test before pushing