Bug 11496

Summary: The number of tested records when searching for duplicates during import is too low
Product: Koha Reporter: Matthias Meusburger <matthias.meusburger>
Component: CatalogingAssignee: Matthias Meusburger <matthias.meusburger>
Status: ASSIGNED --- QA Contact: Testopia <testopia>
Severity: normal    
Priority: P5 - low CC: jonathan.druart, katrin.fischer, liz, m.de.rooy, mtompset
Version: Main   
Hardware: All   
OS: All   
See Also: http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9988
Change sponsored?: --- Patch complexity: Trivial patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Attachments: proposed patch
Bug 11496: Raise number of tested records when searching for duplicate during import.

Description Matthias Meusburger 2014-01-08 09:38:29 UTC
Currently, only the first 10 records returned by koha for a given matchpoint are tested.

This is not enough in some cases, because a significant matching record can be beyond these 10 first results.

Please note that when the checkpoint is about fields with unique identifiers, like ISBN, the problem won't occur.

On the other hand, when the checkpoint is about something less precise, like an author name, the matching record can be beyond the 10 first results.

When that happens, the matchpoint will fail, the match threshold won't be reached, and a duplicate record will be created.
Comment 1 Matthias Meusburger 2014-01-08 09:44:51 UTC Comment hidden (obsolete)
Comment 2 Mark Tompsett 2014-01-28 03:48:05 UTC
Couldn't we have logic that knows about tighter matches (ISBN) vs. less precise matches (authors)? And for the less precise ones, couldn't even 1000 still fail?
Comment 3 Matthias Meusburger 2014-06-17 12:38:48 UTC
There is no need amha to implement such logic, because for tighter matches, even if the limit is high, only a few results will be returned.

ie: having a limit of 1000 will not slow down things for isbn, because only a few isbn's will be returned.

We can safely have a high limit even for tighter matches.

As for your second point, 1000 could possibly fail, but very less likely than with 10.
Comment 4 Mark Tompsett 2015-04-09 15:57:35 UTC
Created attachment 37626 [details] [review]
Bug 11496: Raise number of tested records when searching for duplicate during import.

- Previously, only the first 10 records returned by koha for a given matchpoint were tested.
     This is not enough in many cases, because a significant matching record can be beyond these 10 first results.
     This patch raise the search to 1000 records, for more accurate, but slower results.
Comment 5 Mark Tompsett 2015-04-09 15:58:17 UTC
Rescued this from a 'Patch Does Not Apply' situation. Also, please provide a test plan.
Comment 6 Liz Rea 2015-11-06 01:37:20 UTC
This needs a test plan please, before we can test it.
Comment 7 Katrin Fischer 2020-01-05 12:01:45 UTC
Should this be 'Needs signoff'?