Bug 26543 - Elasticsearch - Importing biblios with multiple items can cause incorrect index
Summary: Elasticsearch - Importing biblios with multiple items can cause incorrect index
Status: BLOCKED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: master
Hardware: All All
: P5 - low major (vote)
Assignee: Nick Clemens
QA Contact:
URL:
Keywords:
Depends on: 25265 26557 26903
Blocks:
  Show dependency treegraph
 
Reported: 2020-09-25 18:29 UTC by Nick Clemens
Modified: 2020-11-03 01:09 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 26543: Refactor import code to reindex once per bib (13.82 KB, patch)
2020-09-25 18:53 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 26543: Unit tests (2.79 KB, patch)
2020-09-25 18:54 UTC, Nick Clemens
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Nick Clemens 2020-09-25 18:29:27 UTC
Similar to bug 25265 we see many request for indexing a single biblio with multiple items attached when importing and matching records

Additionally, the current code repeatedly generates an indexer object, we can reuse the single object to avoid creating multiple connections
Comment 1 Nick Clemens 2020-09-25 18:53:55 UTC
Created attachment 110771 [details] [review]
Bug 26543: Refactor import code to reindex once per bib

To test:
0 - Be using Elasticsearch, apply patches, reindex
1 - Add multiple items to several records in your catalog
2 - Export those records
3 - Update the items on the records in Koha to be 'damaged'
4 - Import the file exported above, matchign on biblionumber and overlaying items
5 - Perform a search to return the imported records
6 - Confirm all items have damaged correctly removed in search results
7 - Import without matching and ignoring items to confirm new records are added
8 - Search to return records and verify they are indexed (shoudl see 2 copies)
9 - Alter the barcodes of the items in koha
10 - Import the file matching on biblionumber and adding items
11 - Search tp return records, confirm the new items are indexed
12 - Export some authority records
13 - Import them without matching and confirm new records are searchable
14 - Alter the existing records, and match on AuthID, you will need to create a matching rule for this
15 - Import and match and confirm the records are searchable and reflect the change back to original values
Comment 2 Nick Clemens 2020-09-25 18:54:01 UTC
Created attachment 110772 [details] [review]
Bug 26543: Unit tests
Comment 3 Andrew Fuerste-Henry 2020-09-25 19:50:54 UTC
I got an error trying to apply this:

Applying: Bug 26543: Refactor import code to reindex once per bib
error: sha1 information is lacking or useless (C4/Biblio.pm).
error: could not build fake ancestor
Patch failed at 0001 Bug 26543: Refactor import code to reindex once per bib
Comment 4 Andrew Fuerste-Henry 2020-09-28 16:43:00 UTC
Applies now that 25265 is pushed!
Comment 5 Andrew Fuerste-Henry 2020-11-02 20:02:30 UTC
Updating test plan. Existing step 9 led to an error as one attempted to create items based on a 952 containing an existing itemnumber: 

To test:
0 - Be using Elasticsearch, apply patches, reindex
1 - Add multiple items to several records in your catalog
2 - Export those records
3 - Update the items on the records in Koha to be 'damaged'
4 - Import the file exported above, matchign on biblionumber and overlaying items
5 - Perform a search to return the imported records
6 - Confirm all items have damaged correctly removed in search results
7 - Import without matching and ignoring items to confirm new records are added
8 - Search to return records and verify they are indexed (shoudl see 2 copies)
9 - Alter exported records to remove itemnumber and barcode
10 - Import the file matching on biblionumber and adding items
11 - Search tp return records, confirm the new items are indexed
12 - Export some authority records
13 - Import them without matching and confirm new records are searchable
14 - Alter the existing records, and match on AuthID, you will need to create a matching rule for this
15 - Import and match and confirm the records are searchable and reflect the change back to original values
Comment 6 Andrew Fuerste-Henry 2020-11-02 20:03:13 UTC
Failing on step 13. New authorities are created but don't come up in searches.
Comment 7 Andrew Fuerste-Henry 2020-11-03 01:09:01 UTC
See blocking bug, 26903. Turns out the behavior I ran into happens on master without this patch.