Bug 21972 - Record matching rule for authorities only works for first 20 authority records
Summary: Record matching rule for authorities only works for first 20 authority records
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: MARC Authority data support (show other bugs)
Version: master
Hardware: All All
: P5 - low major (vote)
Assignee: Bugs List
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-12-07 09:39 UTC by Simon Hohl
Modified: 2024-02-06 14:35 UTC (History)
3 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
MARC File containing geographical authority data (27.48 KB, application/marc)
2018-12-07 09:39 UTC, Simon Hohl
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Simon Hohl 2018-12-07 09:39:00 UTC
Created attachment 82955 [details]
MARC File containing geographical authority data

The summary is already my first interpretation, the following is happening:

I am trying trying to import some custom authority data (see attached mrc file) and have defined a matching rule accordingly:

Match threshold: 1000
Record type: Authority record

Match point 1:
Search index: indentifier-standard
Score: 500
Tag: 024
Subfields: a
Offset/Length: 0

Match point 2:
Search index: indentifier-standard
Score: 500
Tag: 024
Subfields: 2
Offset/Length: 0

While using the command line interface (staging + commit) to automatically update authority data, I noticed that the number of successfully matched records was far too low. 

As a strict test I then did the following:
1. Import the data using the bulkimport script
2. Upload the same data in the staging Interface
3. See which records match and which do not, in order to find a pattern in those who fail.

The only pattern I encountered was the following: Only the (initially) first 20 imported records are actually matched. Meaning only authorities with ID 1 to 20 are successfully matched with the staged import, everything else stands as "No Match". 

I tried this with different sets of place data, still the same result.
Comment 1 Simon Hohl 2018-12-07 12:24:58 UTC
For 1. I should add: Using a fresh Koha install (without any authority data imported yet).
Comment 2 Simon Hohl 2018-12-07 14:30:28 UTC
Some further testing:

1) If I remove the match point for subfield 024$2, the staged records are all matched correctly. But this is not a viable solution for a production environment, because I basically stop checking if the ID in 024$a really is an iDAI.gazetteer ID.
2) I suspect the error is caused by this line:

my ( $authresults, $total ) = $searcher->search_auth_compat( $search_query, 0, 20 );
See: https://gitlab.com/koha-community/Koha/blob/master/C4/Matcher.pm#L718

Is it possible, that the matching rule searches for existing authority records 024$2 subfields with the same value (= iDAI.gazetteer), but the result has a cut off at 20 results? Then, because 'iDAI.gazetteer' is the 024$2 value of ALL place authority records, the result will just contain the first 20 returned and everything else is discarded and the check on 024$a is somehow ignored.

I have never coded anything in Perl, so I am not quite sure how the different rules get applied - I may be wrong.
Comment 3 Simon Hohl 2018-12-12 10:33:36 UTC
I worked around this by creating a composite subfield 024$9 containing both values from $2 and $a. Then I updated the merging rule to only check $9. But I would still consider this a bug.
Comment 4 Martin Renvoize 2019-02-11 17:05:43 UTC
This does indeed feel somewhat wrong to me.. hard limiting results to 20. However, it's not 18.11.x specific as this code is also in master, so I'm marking it as a bug there.
Comment 5 Nick Clemens 2020-10-05 18:22:08 UTC
I think the configurability of that number should be addressed, however, I think the other issue here is the lack of match checks, see bug 26611

Looking the the two matchpoints specified, it seems the second should be a match check - verify the id, then verify it comes from the same source.

The two match points are not searched together, they are separate searches.

It may be more straightforward here to simply specify a single matchpoint that checks:
Tag: 024
Subfields: a2

That would check against both values, rather than needing to search more records, and wouldn't require building the combined index