Currently, only the first 10 records returned by koha for a given matchpoint are tested. This is not enough in some cases, because a significant matching record can be beyond these 10 first results. Please note that when the checkpoint is about fields with unique identifiers, like ISBN, the problem won't occur. On the other hand, when the checkpoint is about something less precise, like an author name, the matching record can be beyond the 10 first results. When that happens, the matchpoint will fail, the match threshold won't be reached, and a duplicate record will be created.
Created attachment 24006 [details] [review] proposed patch Please note that this is a naive implementation of the fix (I've arbitrarily raised the matching records from 10 to 1000). If you agree that this is an issue, a better patch with a configurable amount of records to search might be proposed.
Couldn't we have logic that knows about tighter matches (ISBN) vs. less precise matches (authors)? And for the less precise ones, couldn't even 1000 still fail?
There is no need amha to implement such logic, because for tighter matches, even if the limit is high, only a few results will be returned. ie: having a limit of 1000 will not slow down things for isbn, because only a few isbn's will be returned. We can safely have a high limit even for tighter matches. As for your second point, 1000 could possibly fail, but very less likely than with 10.
Created attachment 37626 [details] [review] Bug 11496: Raise number of tested records when searching for duplicate during import. - Previously, only the first 10 records returned by koha for a given matchpoint were tested. This is not enough in many cases, because a significant matching record can be beyond these 10 first results. This patch raise the search to 1000 records, for more accurate, but slower results.
Rescued this from a 'Patch Does Not Apply' situation. Also, please provide a test plan.
This needs a test plan please, before we can test it.
Should this be 'Needs signoff'?