Created attachment 82955 [details] MARC File containing geographical authority data The summary is already my first interpretation, the following is happening: I am trying trying to import some custom authority data (see attached mrc file) and have defined a matching rule accordingly: Match threshold: 1000 Record type: Authority record Match point 1: Search index: indentifier-standard Score: 500 Tag: 024 Subfields: a Offset/Length: 0 Match point 2: Search index: indentifier-standard Score: 500 Tag: 024 Subfields: 2 Offset/Length: 0 While using the command line interface (staging + commit) to automatically update authority data, I noticed that the number of successfully matched records was far too low. As a strict test I then did the following: 1. Import the data using the bulkimport script 2. Upload the same data in the staging Interface 3. See which records match and which do not, in order to find a pattern in those who fail. The only pattern I encountered was the following: Only the (initially) first 20 imported records are actually matched. Meaning only authorities with ID 1 to 20 are successfully matched with the staged import, everything else stands as "No Match". I tried this with different sets of place data, still the same result.
For 1. I should add: Using a fresh Koha install (without any authority data imported yet).
Some further testing: 1) If I remove the match point for subfield 024$2, the staged records are all matched correctly. But this is not a viable solution for a production environment, because I basically stop checking if the ID in 024$a really is an iDAI.gazetteer ID. 2) I suspect the error is caused by this line: my ( $authresults, $total ) = $searcher->search_auth_compat( $search_query, 0, 20 ); See: https://gitlab.com/koha-community/Koha/blob/master/C4/Matcher.pm#L718 Is it possible, that the matching rule searches for existing authority records 024$2 subfields with the same value (= iDAI.gazetteer), but the result has a cut off at 20 results? Then, because 'iDAI.gazetteer' is the 024$2 value of ALL place authority records, the result will just contain the first 20 returned and everything else is discarded and the check on 024$a is somehow ignored. I have never coded anything in Perl, so I am not quite sure how the different rules get applied - I may be wrong.
I worked around this by creating a composite subfield 024$9 containing both values from $2 and $a. Then I updated the merging rule to only check $9. But I would still consider this a bug.
This does indeed feel somewhat wrong to me.. hard limiting results to 20. However, it's not 18.11.x specific as this code is also in master, so I'm marking it as a bug there.
I think the configurability of that number should be addressed, however, I think the other issue here is the lack of match checks, see bug 26611 Looking the the two matchpoints specified, it seems the second should be a match check - verify the id, then verify it comes from the same source. The two match points are not searched together, they are separate searches. It may be more straightforward here to simply specify a single matchpoint that checks: Tag: 024 Subfields: a2 That would check against both values, rather than needing to search more records, and wouldn't require building the combined index