Bug 16921

Summary:	Authority Linking doesn't match to proper record
Product:	Koha	Reporter:	Nicholas van Oudtshoorn <vanoudt>
Component:	MARC Authority data support	Assignee:	Galen Charlton <gmcharlt>
Status:	CLOSED FIXED	QA Contact:	Testopia <testopia>
Severity:	major
Priority:	P5 - low	CC:	1joynelson, abl, dcook
Version:	16.05
Hardware:	All
OS:	All
See Also:	https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16581 https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13751
Change sponsored?:	---	Patch complexity:	---
Documentation contact:		Documentation submission:
Text to go in the release notes:		Version(s) released in:
Circulation function:
Attachments:	New Linker Module: BestMatch New version of BestMatch linker, that includes using the cache

Description Nicholas van Oudtshoorn 2016-07-14 05:16:31 UTC Comment hidden (obsolete)

Created attachment 53394 [details] [review]
New Linker Module: BestMatch

We've had some trouble here linking our records to their proper authority records. The zebra search seems to be giving back results that match only on the first word. 
Copied from an email to the devel list:
-----
...I'm in the process of trying to relink all of our biblios to the right authority files. Somehow, they were an /incredible/ mess. I've created the authority records okay, but when trying to run link_bibs_to_authorities.pl, it links them to the wrong places.

Investigating, it seems like zebra isn't giving the right search results back.

I can confirm this in yaz-client using the query that is created by koha. Searching for "Bentley, Peter, Ed." returns 4 results, only two of which are close. Actually, the record I'm trying to match is "Bentley, Peter, (Ed.)" - the last option.

Interestingly enough, if I search for "Bentley, 3432531531532513985735981<anything at all here>" it will also show me 
the same four results.So does searching for "Bentley 513513531513515151".But searching for "Bentleyr435435151551251" shows no results, as I would expect.

-----

Not sure what's going on here - but in desperation, I wrote a new Linker Module. It's called the "BestMatch" linker. Basically, it gets all the authorities back from the search, and then searches through them all to find the one that matches perfectly. If there is no perfect match, it tries to find a match that differs only in whitespace/punctuation.

The only problem - it is understandably very SLOW

It's possible that I've completely missed the point here. But the BestMatch linker seems to be doing the job for us.

Note: To do a successful match, the authority data was exported from bibliographic records using koha-auth, then imported into the authorities table, and then the link_bibs_to_authorities.pl script run with the BestMatch linker selected.

Comment 1 Nicholas van Oudtshoorn 2016-07-14 08:08:28 UTC

Created attachment 53396 [details] [review]
New version of BestMatch linker, that includes using the cache

A slight tweak to the BestMatch linker, to use the cache - hopefully giving a slight speed bump!

Comment 2 Jacek Ablewicz 2016-07-14 10:48:35 UTC

(In reply to Nicholas van Oudtshoorn from comment #0)

> Copied from an email to the devel list:
> Investigating, it seems like zebra isn't giving the right search results
> back.
> 
> I can confirm this in yaz-client using the query that is created by koha.
> Searching for "Bentley, Peter, Ed." returns 4 results, only two of which are
> close. Actually, the record I'm trying to match is "Bentley, Peter, (Ed.)" -
> the last option.
> 
> Interestingly enough, if I search for "Bentley,
> 3432531531532513985735981<anything at all here>" it will also show me 
> the same four results.So does searching for "Bentley 513513531513515151".But
> searching for "Bentleyr435435151551251" shows no results, as I would expect.

Hi,
attribute combo (phrase, no trucation, complete field) used in yaz-client test, from the 1st mail in koha-devel list back in June 23:

    @attr 1=Match-heading @attr 4=1 @attr 5=100 @attr 6=3 "Bentley, Peter, Ed."

looks fine to me, and in my test setup it returns a proper result (only one, exact authority record matched). Most likely, problem is not in the Koha authority matching code, but in Zebra itself (it's config, or maybe the particular version you are using is buggy). BTW, do you have ICU tokenizer enabled? If yes, there were some known problems with it (completeness and position attributes not working at all, or not working properly) especially in not up-to-date zebra versions (see Bug 13751 and Bug 16581).

Comment 3 Joy Nelson 2016-07-14 13:55:43 UTC

I also am not seeing any oddities in my search results of authorities, nor any mismatches in the linker script.  Normally the oddities with the linker script are due to setting in the system preferences.   I have heard of other sites with issues with the linker that appeared to be a result of their zebra config.  
-joy


> Hi,
> attribute combo (phrase, no trucation, complete field) used in yaz-client
> test, from the 1st mail in koha-devel list back in June 23:
> 
>     @attr 1=Match-heading @attr 4=1 @attr 5=100 @attr 6=3 "Bentley, Peter,
> Ed."
> 
> looks fine to me, and in my test setup it returns a proper result (only one,
> exact authority record matched). Most likely, problem is not in the Koha
> authority matching code, but in Zebra itself (it's config, or maybe the
> particular version you are using is buggy). BTW, do you have ICU tokenizer
> enabled? If yes, there were some known problems with it (completeness and
> position attributes not working at all, or not working properly) especially
> in not up-to-date zebra versions (see Bug 13751 and Bug 16581).

Comment 4 Nicholas van Oudtshoorn 2016-07-21 07:33:01 UTC

Thanks Joy and Jacek,

Hmmm - I was on 2.0.58 of zebra - updated now, and things seem a lot better.  ICU is enabled - will try with it disabled at some point. However, I'm still having issues. 
For instance, searching for "Young, Robert" returns two results. One of them has a date in $d, the other doesn't. 
Another example, there's a PERSO_NAME of value "T" - and it is there, but koha's returning 104 results (anyone with the intital T in their name, it looks like!).

A more interesting example, searching for the TOPIC_TERM "Theology, Doctrinal" tells me there are three results:
These are:
    Theology, Doctrinal
    Theology, Doctrinal-popular Works
    Theology, Doctrinal.
Not sure what that second one is doing being brought in...

Is this data not particularly nice, and in need of a tidy up? Yep! But I can't understand why the search isn't returning the right results... any ideas?

Comment 5 David Cook 2016-08-01 23:29:03 UTC

(In reply to Nicholas van Oudtshoorn from comment #4)
> Thanks Joy and Jacek,
> 
> Hmmm - I was on 2.0.58 of zebra - updated now, and things seem a lot better.

>     Theology, Doctrinal
>     Theology, Doctrinal-popular Works
>     Theology, Doctrinal.
> Not sure what that second one is doing being brought in...
> 

Nicholas, are you on 2.0.59? If so, that hyphen is what's causing the problem. See bug 16581.

I've been warning the community for a long time that the version of Zebra in Debian is bad when using ICU, but those warnings have mostly been ignored.

Comment 6 David Cook 2016-08-01 23:30:09 UTC

(In reply to Joy Nelson from comment #3)
> I also am not seeing any oddities in my search results of authorities, nor
> any mismatches in the linker script.  Normally the oddities with the linker
> script are due to setting in the system preferences.   I have heard of other
> sites with issues with the linker that appeared to be a result of their
> zebra config.  
> -joy
> 

If I recall correctly, I think Bywater has often shown a preference for using IndexData's repositories for Zebra? So you might have a more up-to-date version of Zebra than Nicholas, I think.

Of course, this comment and my last one are after skimming this bug report for approximately 2 seconds, so I might be misreading :).

Comment 7 Nicholas van Oudtshoorn 2016-08-25 05:44:01 UTC

Hi all! Hmmmm - problems disappeared upgrading to zebra 2.0.62. (Forgot about this bug, actually!)
The version in fedora was a bit older than that - I've updated that too! :-)
Closing this bug - thanks for all the help everyone!